I'm having an issue that I have not encountered before when bulk inserting from a csv file. For whatever reason, the last column isn't being separated on insert. I kept getting type conversion errors that I knew couldn't be true so I changed the datatype to varchar to see what was being inserted. When I looked at the result set, I saw that instead of (ex. 35.44, 56.82 separated in two columns) in the .csv, I saw (ex. 35.44,56.82 all in one column). This of course is why SQL Server was throwing that error, but how can I resolve this. Am I missing something simple?
To sum it up, the Bulk Insert is ignoring the last field terminator and combining the last two columns into one column
My Bulk Insert:
BULK
INSERT [YourTableName]
FROM 'YourFilePathHere'
WITH
(
FIELDTERMINATOR=',',
ROWTERMINATOR = '\n'
)
A row:
YSQ3863,Bag 38x63 YELLOW 50/RL,CS,BAG,17.96,LB,1,50,50,YELLOW,,,,,,63,17.96,,,,38,,2394,,8.15,11.58,19.2,222.41
Related
I want to perform a Bulk Insert for data I get via Stream. Here I get Survey data where each row are the information and answers of a person. I consumed the stream via .net and saved the data row by row each ending with a vblf (I checked if this exist via Word and could see that after each dataset there is a new line). The data are comma separated. First off I created a table with 1000 columns, since I do not know yet how many data will come in but for sure there is not dataset longer than 500 yet and even in the future it will definitly not get longer than 1000 and if so, I can extend the table. Here the Table I created:
The first two datasets looks like this:
"4482359","12526","2014 Company","upload by","new upload","Anonymous","User","anonymous#company.org","","222.222.222.222","1449772662000","undefined","","951071","2015","","2","3","1","5","1","1","3","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","5","1","3","3","3","3","1","2","3","1","3","5","1","","Here ppl can type in some text.","1"
"4482360","12526","2014 Company","upload by","new upload","Anonymous","User","anonymous#company.org","","222.222.222.222","1449772662000","undefined","","951071","2015","","2","5","1","","2","2","3","4","3","1","4","4","4","4","3","3","","4","3","1","4","3","1","4","4","4","3","3","4","4","4","4","3","4","4","4","4","4","4","5","2","3","4","1","3","2","2","5","1","3","","2","","","2"
Now I want to do a Bulk Insert using this command:
USE MyDatabase
BULK INSERT insert_Table FROM 'C:\new.txt'
With (FIRSTROW = 2, FIELDTERMINATOR = ',', ROWTERMINATOR = '\n')
The command runs through and does not thow an error but I get the message 0 rows affected and there is no data in the datatable. Does anyone has an idea what I am doing wrong here?
For some reason I keep receiving the following error when trying to bulk insert a CSV file into SQL Express:
Bulk load data conversion error (type mismatch or invalid character for the
specified codepage) for row 2, column 75 (Delta_SM_RR).
Msg 4864, Level 16, State 1, Line 89
Bulk load data conversion error (type mismatch or invalid character for the
specified codepage) for row 3, column 75 (Delta_SM_RR).
Msg 4864, Level 16, State 1, Line 89
Bulk load data conversion error (type mismatch or invalid character for the
specified codepage) for row 4, column 75 (Delta_SM_RR).
... etc.
I have been attempting to insert this column as both decimal and numeric, and keep receiving this same error (if I take out this column, the same error appears for the subsequent column).
Please see below for an example of the data, all data points within this column contain decimals and are all rounded after the third decimal point:
Delta_SM_RR
168.64
146.17
95.07
79.85
60.52
61.03
-4.11
-59.57
1563.09
354.36
114.78
253.46
451.5
Any sort of help or advice would be greatly appreciated as it seems that a number of people of SO have come across this issue. Also, if anyone knows of another automated way to load a CSV into SSMS, that would be a great help as well.
Edits:
Create Table Example_Table
(
[Col_1] varchar(255),
[Col_2] numeric(10,5),
[Col_3] numeric(10,5),
[Col_4] numeric(10,5),
[Col_5] date,
[Delta_SM_RR] numeric(10,5),
)
GO
BULK INSERT
Example_Table
FROM 'C:\pathway\file.csv'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n',
FIRSTROW = 2
);
Table Schema - This is a standalone table (further calculations and additional tables are built off of this single table, however at the time of bulk insert it is the only table)
It's likely that your data has an error in it. That is, that there is a character or value that can't be converted explicitly to NUMERIC or DECIMAL. One way to check this and fix it is to
Change [Delta_SM_RR] numeric(10,5) to [Delta_SM_RR] nvarchar(256)
Run the bulk insert
Find your error row: select * from Example_Table where [Delta_SM_RR] like '%[^-.0-9]%'
Fix the data at the source, or delete from Example_Table where [Delta_SM_RR] like '%[^-.0-9]%'
The last statements returns/deletes rows where there is something other than a digit, period, or hyphen.
For your date column you can follow the same logic above, by changing the column to VARCHAR, and then find your error by using ISDATE() to find the ones which can't be converted.
I'll bet anything there is some weird character in your data set. Open your data set in Notepad++ and view the data. Any aberration should become apparent very quickly! The problem is coming from Col75 and it's affecting the first several rows, and thus everything that comes after that also fails to load.
Make sure that .csv is not using text qualifiers and that none of your fields in the .csv have a comma inside the desired value.
I am struggling with this issue right now. The issue is that I have a 68 column report I am trying to import.
Column 17 is a "Description" column that has a double quote text qualifier on top of the comma delimitation.
Bulk insert with a comma field terminator won't identify the double quote text qualifier and munge all of the data to the right of the offending column.
It looks like to overcome this, you need to create a .fmt file to instruct the Bulk Insert which columns it needs to treat as simple delimited, and which columns it needs to treat as delimited and qualified (see this answer).
I've got the simple table in CSV format:
999,"01/01/2001","01/01/2001","7777777","company","channel","01/01/2001"
990,"01/01/2001","01/01/2001","767676","hhh","tender","01/01/2001"
3838,"01/01/2001","01/01/2001","888","jhkh","jhkjh","01/01/2001"
08987,"01/01/2001","01/01/2001","888888","hkjhjkhv","jhgjh","01/01/2001"
8987,"01/01/2001","01/01/2001","9999","jghg","hjghg","01/01/2001"
jhkjhj,"01/01/2001","01/01/2001","9999","01.01.2001","hjhh","01/01/2001"
090009,"","","77777","","","01/01/2001"
980989,"01/01/2001","01/01/2001","888","","jhkh","01/01/2001"
0000,"01/01/2001","01/01/2001","99999","jhjh","","01/01/2001"
92929,"01/01/2001","01/01/2001","222","","","01/01/2001"
I'm trying to import that data into SQL Server using BULK INSERT (Transact-SQL)
set dateformat DMY;
BULK INSERT Oracleload
FROM '\\Mac\Home\Desktop\Test\T_DOGOVOR.csv'
WITH
(FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n',
KEEPNULLS);
On the output I get the next error:
Msg 4864, Level 16, State 1, Line 4
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 2 (date_begin)....
Something wrong with date format maybe. But what script I need to write to fix that error?
Please help.
Thanks in advance.
BULK INSERT (nor bcp) cannot (properly) handle CSV files, specially if they have (correctly) " quotes. Alternatives are SSIS or PowerShell.
I always look at the data in Notepad++ to see if there are some weird characters, or non-printable characters, like a line break or something. For this, it seems like you can open it using Notepad (if you don't have Notepad++) do a find-replace for " to nothing... Save the file, and re-do the Bulk Load.
This record:
jhkjhj,"01/01/2001","01/01/2001","9999","01.01.2001","hjhh","01/01/2001"
The first column has a numeric type of some kind. You can't put the jhkjhj value into that field.
Additionally, some records have empty values ("") in date fields. These are likely to be to interpreted as empty strings, rather than null dates, and not convert properly.
But the error refers to "row 1, column 2". That's this value:
"01/01/2001"
Again, the import is interpreting this as a string, rather than a date. I suspect it's trying to import the quotes (") instead of just using them as separators.
You might try bulk loading to a special holding table, and then re-importing from there. Alternatively, you can change how data is exported or write a program to pre-clean it — strip the quotes from fields that shouldn't have them, isolate records that have data that won't insert to an exception file and report.
Working in the Visual Studio database feature. I've got two tables and I need to load a .csv file into them. I broke out the .csv file into my first and 2nd table. I'm trying a bulk insert
BULK INSERT Course
FROM 'E:\CourseInfo.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
TABLOCK
)
Everything seems right to me but I receive an error saying:
Msg 4864, Level 16, State 1, Line 1
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 2, column 3 (ID).
Here is a snippet of my .CSV file being used.
CourseID,CourseTitle,ID,
AC107,Accounting I,1,
AC107,Accounting I,2,
AC110,Payroll Accounting,3,
AC212,Taxation I,4,
AC212,Taxation I,5,
What is meant by mismatch or invalid char? I've tried removing all the values for column 'ID' but that still rendered the same error. I had ID set to auto-increment setting it as "isEntity" but have tried both ways with it set to true and false, still same error.
Possible error: Laid a huge egg, think I have to normalize this prior to doing this because there are multiples of the same class with different ID and that isn't right.
To Answer my own question, I had my PK set to ID and it was showing up for Duplicate Rows where there was duplicate data. I have to remove duplicate courseID, and CourseTitle rows prior to populating my ID Column with values.
How can I bulk insert a file like below?
test.txt
012341231
013212313
011312321
012312312
The text file does not contain a delimiter. I have used:
BULK INSERT tbl_import_#id#
FROM '../test.txt'
WITH
(FIELDTERMINATOR = '\t',
ROWTERMINATOR = '\n')
and I got an error for that. Appreciate any help thanks.
There is no problem. You can specify a field terminator even if your file doesn't have any field terminators like \t or ,.
Please try to post what error you have got. Check your FROM file ".../test.txt" location and table schema to import data. Better to post your error. I cannot reproduce your error. It works fine for me (I used your values).
Just run the query without FILEDTERMINATOR
BULK INSERT tbl_import_#id#
FROM '../test.txt'
WITH (ROWTERMINATOR = '\n')
The FIELDTERMINATOR argument would be helpful in case you had multiple columns in your table (more values per row). But I can see that this is not the case, so you don't need to separate values except by rows, which will be records in your table.
EDIT:
In case you can use a different table, just create a table with only 1 column(ID column) and run the import (the query above).
After that, run an ALTER command and add the other columns that you want.