Bulk Insert does not insert data - sql-server
I want to perform a Bulk Insert for data I get via Stream. Here I get Survey data where each row are the information and answers of a person. I consumed the stream via .net and saved the data row by row each ending with a vblf (I checked if this exist via Word and could see that after each dataset there is a new line). The data are comma separated. First off I created a table with 1000 columns, since I do not know yet how many data will come in but for sure there is not dataset longer than 500 yet and even in the future it will definitly not get longer than 1000 and if so, I can extend the table. Here the Table I created:
The first two datasets looks like this:
"4482359","12526","2014 Company","upload by","new upload","Anonymous","User","anonymous#company.org","","222.222.222.222","1449772662000","undefined","","951071","2015","","2","3","1","5","1","1","3","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","1","3","3","3","3","1","2","3","1","3","5","1","","Here ppl can type in some text.","1"
"4482360","12526","2014 Company","upload by","new upload","Anonymous","User","anonymous#company.org","","222.222.222.222","1449772662000","undefined","","951071","2015","","2","5","1","","2","2","3","4","3","1","4","4","4","4","3","3","","4","3","1","4","3","1","4","4","4","3","3","4","4","4","4","3","4","4","4","4","4","4","5","2","3","4","1","3","2","2","5","1","3","","2","","","2"
Now I want to do a Bulk Insert using this command:
USE MyDatabase
BULK INSERT insert_Table FROM 'C:\new.txt'
With (FIRSTROW = 2, FIELDTERMINATOR = ',', ROWTERMINATOR = '\n')
The command runs through and does not thow an error but I get the message 0 rows affected and there is no data in the datatable. Does anyone has an idea what I am doing wrong here?
Related
Import data into SQL Server using BCP utility (export the log file with the error records and continue inserting with the normal records)
I have a data set and want to import it into my database with the condition: In case there is a record that cannot be imported, it can be extracted into a log Although existing records can not be imported but still allow import of records that can be imported (other records) and continue to process Currently I use the BCP utility to import data into the table from the csv file with: bcp table_name IN C:\Users\09204086121\Desktop\data.csv -T -c -o C:\Users\09204086121\Desktop\logOut.log -e C:\Users\09204086121\Desktop\errOut.log It just satisfies my condition 1 above. I need that when the record has error (duplicate primary key,...), write to log (1) and continue to insert into the table the other normal records (2). I came up with the idea that combining trigger with bcp, after creating a trigger and adding the parameter -h "FIRE_TRIGGERS" to the bcp statement, the insert will ignore records that have the same key but it won't write to the log. This is my trigger. ALTER TRIGGER [PKGORDERCOMMON].[T_ImportData] ON [PKGORDERCOMMON].[IF_R_BUNRUI1] INSTEAD OF INSERT AS BEGIN --Insert non duplicate records INSERT INTO [IF_R_BUNRUI1] ( SYSTEM_KB, BUNRUI1_CD, BUNRUI1_KANJI_NA, BUNRUI1_KANA_NA, CREATE_TS ) SELECT SYSTEM_KB, BUNRUI1_CD, BUNRUI1_KANJI_NA, BUNRUI1_KANA_NA, CREATE_TS FROM inserted i WHERE NOT EXISTS ( SELECT * FROM [IF_R_BUNRUI1] c WHERE c.BUNRUI1_CD = i.BUNRUI1_CD AND c.SYSTEM_KB = i.SYSTEM_KB ); END; Is there anyone who can help me.
BCP is not meant for what you are asking it to do (separate good and bad records). For instance, bcp -e option has a limit to how many records it will show. Im not sure if this limit is tied to the "max errors" option, but regardless there is a limit. Your best option is to load all the records and address bad data in t-sql. Load all records in such a way to ignore conversion errors. Either: load entire line from file into a single, large varchar column. Parse out columns and qc data as needed. or load all columns from source file into generic varchar columns with large enough size to accomodate your source data. Either way, when done, use t-sql to inspect your data and split among good/bad records.
SQL Server Bulk Insert CSV Issue
I'm having an issue that I have not encountered before when bulk inserting from a csv file. For whatever reason, the last column isn't being separated on insert. I kept getting type conversion errors that I knew couldn't be true so I changed the datatype to varchar to see what was being inserted. When I looked at the result set, I saw that instead of (ex. 35.44, 56.82 separated in two columns) in the .csv, I saw (ex. 35.44,56.82 all in one column). This of course is why SQL Server was throwing that error, but how can I resolve this. Am I missing something simple? To sum it up, the Bulk Insert is ignoring the last field terminator and combining the last two columns into one column My Bulk Insert: BULK INSERT [YourTableName] FROM 'YourFilePathHere' WITH ( FIELDTERMINATOR=',', ROWTERMINATOR = '\n' ) A row: YSQ3863,Bag 38x63 YELLOW 50/RL,CS,BAG,17.96,LB,1,50,50,YELLOW,,,,,,63,17.96,,,,38,,2394,,8.15,11.58,19.2,222.41
Data import and processing - CSV
I have a number of data sources stored as CSVs and a database table structure in 3NF on SQL Server 2008r2. Some of the columns within the data sources are unrequired. What I would like to do is to insert the data from each data source into a separate temporary table where data types (and ideally column names) are declared based on the data being inserted. If this is not possible importing the data as the char datatype would help. From here I would like to select the relevant data into my standard database structure and once complete drop the temporary tables. I think I will have to use some cast statements to ensure the data is in the right format. I am happy to write a stored procedure to do this although the data is published periodically and occasionally the structure is changed so it will require some manual work. I have tried to do this with a BULK INSERT but I receive Msg 208 regarding an invalid object name for the temporary table. USE db1; GO BULK INSERT #TempDetails FROM 'C:\Documents\Data\Text Files\Details.txt' WITH ( DATAFILETYPE = 'char' , FIELDTERMINATOR = ',' , ROWTERMINATOR = '\n' ); Returns Msg 208, Level 16, State 82, Line 2 Invalid object name '#TempDetails'. What is the best way for me to achieve this? My fall back alternative is to import the CSVs into Access and then import that into SQL for the processing but I would like this in code so it is easy to replicate.
Bulk INSERT without FIELDDELIMITER
How can I bulk insert a file like below? test.txt 012341231 013212313 011312321 012312312 The text file does not contain a delimiter. I have used: BULK INSERT tbl_import_#id# FROM '../test.txt' WITH (FIELDTERMINATOR = '\t', ROWTERMINATOR = '\n') and I got an error for that. Appreciate any help thanks.
There is no problem. You can specify a field terminator even if your file doesn't have any field terminators like \t or ,. Please try to post what error you have got. Check your FROM file ".../test.txt" location and table schema to import data. Better to post your error. I cannot reproduce your error. It works fine for me (I used your values).
Just run the query without FILEDTERMINATOR BULK INSERT tbl_import_#id# FROM '../test.txt' WITH (ROWTERMINATOR = '\n') The FIELDTERMINATOR argument would be helpful in case you had multiple columns in your table (more values per row). But I can see that this is not the case, so you don't need to separate values except by rows, which will be records in your table. EDIT: In case you can use a different table, just create a table with only 1 column(ID column) and run the import (the query above). After that, run an ALTER command and add the other columns that you want.
Is SQL Server Bulk Insert Transactional?
If I run the following query in SQL Server 2000 Query Analyzer: BULK INSERT OurTable FROM 'c:\OurTable.txt' WITH (CODEPAGE = 'RAW', DATAFILETYPE = 'char', FIELDTERMINATOR = '\t', ROWS_PER_BATCH = 10000, TABLOCK) On a text file that conforms to OurTable's schema for 40 lines, but then changes format for the last 20 lines (lets say the last 20 lines have fewer fields), I receive an error. However, the first 40 lines are committed to the table. Is there something about the way I'm calling Bulk Insert that makes it not be transactional, or do I need to do something explicit to force it to rollback on failure?
BULK INSERT acts as a series of individual INSERT statements and thus, if the job fails, it doesn't roll back all of the committed inserts. It can, however, be placed within a transaction so you could do something like this: BEGIN TRANSACTION BEGIN TRY BULK INSERT OurTable FROM 'c:\OurTable.txt' WITH (CODEPAGE = 'RAW', DATAFILETYPE = 'char', FIELDTERMINATOR = '\t', ROWS_PER_BATCH = 10000, TABLOCK) COMMIT TRANSACTION END TRY BEGIN CATCH ROLLBACK TRANSACTION END CATCH
You can rollback the inserts . To do that we need to understand two things first BatchSize : No of rows to be inserted per transaction . The Default is entire Data File. So a data file is in transaction Say you have a text file which has 10 rows and row 8 and Row 7 has some invalid details . When you Bulk Insert the file without specifying or with specifying batch Size , 8 out of 10 get inserted into the table. The Invalid Row's i.e. 8th and 7th gets failed and doesn't get inserted. This Happens because the Default MAXERRORS count is 10 per transaction. As Per MSDN : MAXERRORS : Specifies the maximum number of syntax errors allowed in the data before the bulk-import operation is canceled. Each row that cannot be imported by the bulk-import operation is ignored and counted as one error. If max_errors is not specified, the default is 10. So Inorder to fail all the 10 rows even if one is invalid we need to set MAXERRORS=1 and BatchSize=1 Here the number of BatchSize also matters. If you specify BatchSize and the invalid row is inside the particular batch , it will rollback the particular batch only ,not the entire data set. So be careful while choosing this option Hope this solves the issue.
As stated in the BATCHSIZE definition for BULK INSERT in MSDN Library (http://msdn.microsoft.com/en-us/library/ms188365(v=sql.105).aspx) : "If this fails, SQL Server commits or rolls back the transaction for every batch..." In conclusion is not necessary to add transactionality to Bulk Insert.
Try to put it inside user-defined transaction and see what happens. Actually it should roll-back as you described it.