I have a parent package in SSIS that has for each loop container that executes 2 child packages in a sequential manner. The connection values are passed to the child packages as parameters. The child packages dynamically connect to a flat-file source and a derived column is being used to replace the "\N" values with NULL and it is finally loaded in the SQL server destination. The total size of the flat file source is 3GB. The for-each loop executes in the first iteration, but it is failing in the second iteration. I am getting an error message of buffer allocation failed.
Error message
The system reports an 85 percent memory load. There are 32767590400 bytes of physical memory with 4636983296 bytes free. There are 4294836224 bytes of virtual memory with 362987520 bytes free. The paging file has 43437899776 bytes with 13638811648 bytes free.
The Data Flow task failed to create a buffer to call PrimeOutput for output "Flat File Source" (23) on component "Flat File Source Output" (27). This error usually occurs due to an out-of-memory condition.
Memory pressure was alleviated, buffer manager is not throttling allocations anymore
(in the buffer tuning log)
I have changed the default buffer size from 10 MB to 50 MB and default buffer max rows from 10000 to 50000. also changed the rows per batch to 10000 for files larger than 100MB.
still, I am facing the same issue. can anyone help me resolve this issue?
thanks,
Abhishek
I would suggest setting the AutoAdjustBufferSize to True.
Then you need to calculate the max size of the row of the table.
You can use this SQL where you replace your table and a schema in the quotes:
SELECT
SUM (max_length) [row_length]
,104857600.0/SUM (max_length) [Max_100MB_Buffer_Rows]
,2147483647.0/SUM (max_length) [Max_2GB_Buffer_Rows]
FROM sys.tables t
JOIN sys.columns c ON t.object_id=c.object_id
JOIN sys.schemas s ON t.schema_id=s.schema_id
WHERE t.name = '{my_table_name}' AND s.name = '{my_schema_name}'
Then depending on which version your packages are (2015 or 2017) you either use 100MB buffer (2015) or 2GB buffer (2017).
To be on the safe side use 100MB buffer.
Basically, you need to set a good Max num rows (not too high) depending on how wide your data is.
According to this explanation[1] how it is possible to recover deleted records from an SQLite-database, two things happen when you delete a record: The area containing the deleted record is added to the free list (as a free block) and the "header" in the content area is overwritten.
The content itself is not (or only a few bytes) overwritten, so e.g. by opening the database file with a hex editor you can still see parts of the deleted record.
So to check whether this is true or not, here’s a minimal example:
sqlite3 example.db
sqlite> CREATE TABLE 'test'(aColumn VARCHAR(25));
sqlite> INSERT INTO test VALUES('AAAAAAAAAA');
sqlite> INSERT INTO test VALUES('BBBBBBBBBB');
sqlite> INSERT INTO test VALUES('CCCCCCCCCC');
sqlite> DELETE FROM test WHERE aColumn = 'BBBBBBBBBB';
What I’m expecting now is that – opening the file with a hex editor – between CCCC… and AAAA… there are still some Bs (but some bytes before the Bs or the first few Bs should have been overwritten).
What really happens is that the whole area with Bs has been overwritten with nullbytes.
This happens with larger databases, too. The content area is always cleaned with nullbytes and because of this it is impossible to recover anything.
Is there any option or flag which I have to use to get the results shown in the blogpost?
I need to create my SQLite-databases in that way, to be able to recover deleted records (or parts of them) with forensic tools.
[1]http://sandersonforensics.com/forum/content.php?222-Recovering-deleted-records-from-an-SQLite-database
Your version of SQLite happens to be compiled with SQLITE_SECURE_DELETE enabled.
I'm getting this error:
ORA-1691: unable to extend lobsegment ABC.SYS_LOB0014859757C00018$$ by 1280 in tablespace ABC
The tablespace is build like the folowing:
CREATE TABLESPACE "ABC" DATAFILE
'/ora/db/user/abc1.db' SIZE 4194304000,
'/ora/db/user/abc2.db' SIZE 4194304000,
'/ora/db/user/abc3.db' SIZE 4194304000,
'/ora/db/user/abc4.db' SIZE 4194304000
LOGGING ONLINE PERMANENT BLOCKSIZE 8192
EXTENT MANAGEMENT LOCAL UNIFORM SIZE 10485760 SEGMENT SPACE MANAGEMENT AUTO
How can I extend the tablespace? Do I need to restart db after extending?
You can extend a tablespace by adding an additional datafile to it or by extending an existing one. Since you currently seem to have a convention of uniformly sized files, I'd just add another one:
ALTER TABLESPACE "ABC" ADD DATAFILE '/ora/db/user/abc5.db' SIZE 4194304000;
This can be done with the database and tablespace online, and there's no need to restart anything.
I have an extremely large database and most of the space is the index size. I moved several indexes to a different file group (just to experiment) but no matter what I do I cannot reduce the size of the MDF.
I tried shrink database, shrink files, rebuilding clustered index. What can I do to reclaim that space in the MDF? I've moved 25GB worth of indexes to a different file group. Is it even possible to reduce my mdf by that same 25gb (or close to it)?
SQL Server 2008 Enterprise
Group Total Space Avail Space
PRIMARY 388485.000000 27126.3125000
Index 24778.375000 26.6250000
Options I tried for shrink:
TSQL:
1. DBCC SHRINKFILE (1, 10);
2. DBCC SHRINKFILE (1, TRUNCATE_ONLY);
UI: Shrink database:
1. Without 'reorganize files...'
2. With 'reorganize files...'
2a. set max free space = 10%
2b. set max free space = 90%
2c. set max free space = 75%
UI: Shrink files: (type =data, group = primary, filename = db name)
1. Release unused space
2. reorganize pages... setting shrink file = 2GB (max it allows)
3. reorganize pages... setting shrink file = 361,360 (min allowed)
did not try 'Empty file' option because that doesnt seem like what i want.
I've noticed in the past that shrinking the data file in smaller chunks can be more effective than trying to shrink it all in one go. If you were to attempt to use a similar strategy then you'd want to do something like below:
DECLARE #targetSize AS INT = 388000;
DECLARE #desiredFinalSize AS INT = 362000;
DECLARE #increment AS INT = 300;
DECLARE #sql AS VARCHAR(200);
WHILE #targetSize > #desiredFinalSize
BEGIN
SET #sql = 'DBCC SHRINKFILE(''MyDataFileName'', ' + CAST(#targetSize AS VARCHAR(10)) + ');'
SELECT #sql;
EXEC(#sql);
SET #targetSize = #targetSize - #increment;
END
OK, try through the UI a "Shrink Database" with "0%" max free space.
The 10% setting you used before "worked" but your 27 gigs free space is under 10% (its like 7%) of your total DB.
Bear in mind this may cause you performance issues down the road since your DB will need to grow again, and this will also cause performance issues now since shrinking leads to fragmentation which is bad.
However if space is your primary concern the above method should work.
This step looks wrong
UI: Shrink files: (type =data, group = primary, filename = db name)
1. Release unused space
2. reorganize pages... setting shrink file = 2GB (max it allows)
3. reorganize pages... setting shrink file = 361,360 (min allowed)
Among your question steps, this is the right one.
Shrink File
File type (Data), group (PRIMARY) Name ()
Shrink Action = reorganize pages (=move to front), Shrink File *
Shrink File is entered in MB, so a value of 361,360 would be a third of a Terabyte. The easiest way to set it to minimum is to enter 0, tab out -> it will replace with the min allowed.
Right now we need to get the size down to something manageable for backups
This is completely the wrong way to go about it. BACKUPs do not include free space, so whether your primary mdf file has 100GB of free space or 1TB, as long as the data resides on 200GB, that is the size of the backup files. For SQL Server 2008, you can use "with compression", which gives an average size of about 1/6th.
sometimes i've noticed that if i shrink down a file, it won't shrink below the initial size set on the file. could you go see what the initial file size of the file is?
(please read the update section below, I leave the original question too for clarity)
I am inserting many files into a SQL Server db configured for filestream.
I am inserting in a loop the files from a folder to a database table.
Everything goes fine until I try to insert a 600 MB file.
As it inserts it there is a +600MB memory usage in task manager and I have the error.
The DB size is < 1 GB and the total size of documents is 8 GB, I am using SQL Server Express R2, and according to the documentation I could have problems only if trying to insert a document that is greater than 10 GB (Express limitation) - Current DB Size.
Can anyone tell me why do I have this error? It is very crucial for me.
UPDATE FOR BOUNTY:
I offered 150 because it is very crucial for me!
This seems to be a limitation of Delphi memory Manager, trying to insert a document bigger than 500MB, I didn't check the exact threshold anyway it is between 500 and 600MB). I use SDAC components, in particular a TMSQuery (but I think the same can be done with and TDataset descendant), to insert the document in a table that has a PK (ID_DOC_FILE) and a varbinary(max) field (DOCUMENT) I do:
procedure UploadBigFile;
var
sFilePath: String;
begin
sFilePath := 'D:\Test\VeryBigFile.dat';
sqlInsertDoc.ParamByName('ID_DOC_FILE').AsInteger := 1;
sqlInsertDoc.ParamByName('DOCUMENT').LoadFromFile(sFilePath, ftblob);
sqlInsertDoc.Execute;
sqlInsertDoc.Close;
end;
SDAC team told me this is a limitation of Delphi memory manager. Now since SDAC doesn't support filestream I cannot do what has been suggested in c# in the first answer. Is the only solution reporting to Embarcadero and ask a bug fix?
FINAL UPDATE:
Thanks, really, to all you that answered me. For sure inserting big blobs can be a problem for the Express Edition (because the limitations of 1 GB of ram), anyway I had the error on the Enterprise edition, and it was a "delphi" error, not a sql server one. So I think that the answer that I accepted really hits the problem, even if I have no time to verify it now.
SDAC team told me this is a limitation of Delphi memory manager
To me that looked like an simplistic answer, and I investigated. I don't have the SDAC components and I also don't use SQL Server, my favorites are Firebird SQL and the IBX component set. I tried inserting an 600Mb blob into a table, using IBX, then tried the same using ADO (covering two connection technologies, both TDataSet descendants). I discovered the truth is somewhere in the middle, it's not really the memory manager, it's not SDAC's fault (well... they are in a position to do something about it, if many more people attempt inserting 600 Mb blobs into databases, but that's irrelevant to this discussion). The "problem" is with the DB code in Delphi. As it turns out Delphi insists on using an single Variant to hold whatever type of data one might load into an parameter. And it makes sense, after all we can load lots of different things into an parameter for an INSERT. The second problem is, Delphi wants to treat that Variant like an VALUE type: It copies it around at list twice and maybe three times! The first copy is made right when the parameter is loaded from the file. The second copy is made when the parameter is prepared to be sent to the database engine.
Writing this is easy:
var V1, V2:Variant;
V1 := V2;
and works just fine for Integer and Date and small Strings, but when V2 is an 600 Mb Variant array that assignment apparently makes a full copy! Now think about the memory space available for a 32 bit application that's not running in "3G" mode. Only 2 Gb of addressing space are available. Some of that space is reserved, some of that space is used for the executable itself, then there are the libraries, then there's some space reserved for the memory manager. After making the first 600 Mb allocation there just might not be enough available addressing space to allocate an other 600 Mb buffer! Because of this it's safe to blame it on the memory manager, but then again, why exactly does the DB stuff need an other copy of the 600 Mb monster?
One possible fix
Try splitting up the file into smaller, more manageable chunks. Set up the database table to have 3 fields: ID_DOCUMENT, SEQUENCE, DOCUMENT. Also make the primary key on the table to be (ID_DOCUMENT, SEQUENCE). Next try this:
procedure UploadBigFile(id_doc:Integer; sFilePath: String);
var FS:TFileStream;
MS:TMemoryStream;
AvailableSize, ReadNow:Int64;
Sequence:Integer;
const MaxPerSequence = 10 * 1024 * 1024; // 10 Mb
begin
FS := TFileStream.Create(sFilePath, fmOpenRead);
try
AvailableSize := FS.Size;
Sequence := 0;
while AvailableSize > 0 do
begin
if AvailableSize > MaxPerSequence then
begin
ReadNow := MaxPerSequence;
Dec(AvailableSize, MaxPerSequence);
end
else
begin
ReadNow := AvailableSize;
AvailableSize := 0;
end;
Inc(Sequence); // Prep sequence; First sequence into DB will be "1"
MS := TMemoryStream.Create;
try
MS.CopyFrom(FS, ReadNow);
sqlInsertDoc.ParamByName('ID_DOC_FILE').AsInteger := id_doc;
sqlInsertDoc.ParamByName('SEQUENCE').AsInteger := sequence;
sqlInsertDoc.ParamByName('DOCUMENT').LoadFromStream(MS, ftblob);
sqlInsertDoc.Execute;
finally MS.Free;
end;
end;
finally FS.Free;
end;
sqlInsertDoc.Close;
end;
You could loop through the byte stream of the object you are trying to insert and essentially buffer a piece of it at a time into your database until you have your entire object stored.
I would take a look at the Buffer.BlockCopy() method if you're using .NET
Off the top of my head, the method to parse your file could look something like this:
var file = new FileStream(#"c:\file.exe");
byte[] fileStream;
byte[] buffer = new byte[100];
file.Write(fileStream, 0, fileStream.Length);
for (int i = 0; i < fileStream.Length; i += 100)
{
Buffer.BlockCopy(fileStream, i, buffer, 0, 100);
// Do database processing
}
Here is an example that reads a disk file and saves it into a FILESTREAM column. (It assumes that you already have the transaction Context and FilePath in variables "filepath" and "txContext".
'Open the FILESTREAM data file for writing
Dim fs As New SqlFileStream(filePath, txContext, FileAccess.Write)
'Open the source file for reading
Dim localFile As New FileStream("C:\temp\microsoftmouse.jpg",
FileMode.Open,
FileAccess.Read)
'Start transferring data from the source file to FILESTREAM data file
Dim bw As New BinaryWriter(fs)
Const bufferSize As Integer = 4096
Dim buffer As Byte() = New Byte(bufferSize) {}
Dim bytes As Integer = localFile.Read(buffer, 0, bufferSize)
While bytes > 0
bw.Write(buffer, 0, bytes)
bw.Flush()
bytes = localFile.Read(buffer, 0, bufferSize)
End While
'Close the files
bw.Close()
localFile.Close()
fs.Close()
You're probably running into memory fragmentation issues somewhere. Playing around with really large blocks of memory, especially in any situation where they might need to be reallocated tends to cause out of memory errors when in theory you have enough memory to do the job. If it needs a 600mb block and it can't find a hole that's 600mb wide that's it, out of memory.
While I have never tried it my inclination for a workaround would be to create a very minimal program that does ONLY the one operation. Keep it absolutely as simple as possible to keep the memory allocation minimal. When faced with a risky operation like this call the external program to do the job. The program runs, does the one operation and exits. The point is the new program is in it's own address space.
The only true fix is 64 bit and we don't have that option yet.
I recently experienced a similar problem while running DBCC CHECKDB on a very large table. I would get this error:
There is insufficient system memory in
resource pool 'internal' to run this
query.
This was on SQL Server 2008 R2 Express. The interesting thing was that I could control the occurrence of the error by adding or deleting a certain number of rows to the table.
After extensive research and discussions with various SQL Server experts, I came to the conclusion that the problem was a combination of memory pressure and the 1 GB memory limitation of SQL Server Express.
The recommendation given to me was to either
Acquire a machine with more memory and a
licensed edition of SQL Server or...
Partition the table into sizeable
chunks that DBCC CHECKDB could
handle
Due the complicated nature of parsing these files into the FILSTREAM object, I would recommend the filesystem method and and simply use SQL Server to store the locations of the files.
"While there are no limitations on the number of databases or users supported, it is limited to using one processor, 1 GB memory and 4 GB database files (10 GB database files from SQL Server Express 2008 R2)." It is not the size of the database files that is an insue but "1 GB memory". Try spitting the 600MB+ file but putting it in the stream.