I have one problem with bulk insert. I have about 7 million records in the csv file, when I make the following query, just half of the total records are entered in the table.
The Query I use:
BULK INSERT *Table_Name* FROM 'file path'
WITH
(
FIRSTROW = 1, -- csv dont have header
FIELDTERMINATOR = '0x0a',
ROWTERMINATOR = '\n',
ERRORFILE = '*error file path*',
TABLOCK
)
Table Creation query:
CREATE TABLE *Table_Name*
(
ID int NOT NULL IDENTITY(1,1) PRIMARY KEY,
*Column_Name* varchar(50)
);
Where I make a mistake?
How to load all records from a csv file (all 7 million)?
Related
I have a CSV file in UTF8 encoding and I would like to import data into SQL Server DB table.
In some cells I have stored values like:
±40%;
16.5±10%;
All columns load perfectly but only columns with ± character show in DB this:
All columns where I would like to store this character I use nvarchar(50) with collation Latin1_General_100_CS_AS_WS_SC_UTF8
Is there any wait how this character store into DB ?
Thank you
EDIT
I use for load CSV file this:
BULK INSERT [dbo].[x]
FROM 'c:\Users\x\Downloads\x.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ';', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'c:\Users\x\Downloads\xx.csv',
TABLOCK
);
I also try to change SSMS Options:
‘Tools‘ -> ‘Options‘ -> ‘Environment‘ -> ‘Fonts and Colors‘ -> Select ‘Grid Results’
Set Font to Arial but without positive results
I have over 20 million records in many files, which I want to import
Have you tried adding a CODEPAGE=65001 (UTF-8) to the WITH clause of the BULK INSERT statement?
BULK INSERT [dbo].[x]
FROM 'c:\Users\x\Downloads\x.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ';', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'c:\Users\x\Downloads\xx.csv',
CODEPAGE = 65001,
TABLOCK
);
I need to read data from .csv file which contains many records, but in last row
there is string "the_end" and after that there is no LF sign.
Here is my csv file:
1,James,Smith,19750101
2,Meggie,Smith,19790122
3,Robert,Smith,20071101
4,Alex,Smith,20040202
the_end
Below my sql script which reads data into temporary table:
create table #Data
(
id int,
first_name varchar(50),
last_name varchar(50),
birthdate smalldatetime
)
bulk insert #Data
from 'C:\Users\Michał\Desktop\csvtest.csv'
with
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
select * from #Data
drop table #Data
Error is:
Msg 4832, Level 16, State 1, Line 13
Bulk load: An unexpected end of file was encountered in the data file.
Msg 7399, Level 16, State 1, Line 13
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 13
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
I want to read all data into my table without this last line label "the_end", that's obvious. How can i manage that? I cannot modify this file, because it's given from outside and I cannot do that.
Use the lastrow option
bulk insert #Data
from 'C:\Users\Michał\Desktop\csvtest.csv'
with
(
FIELDTERMINATOR = ','
, ROWTERMINATOR = '\n'
, lastrow = 4
)
dynamic sql example using a variable for lastline:
create table #Data (
id int
, first_name varchar(50)
, last_name varchar(50)
, birthdate smalldatetime
);
declare #lastline int = 4;
declare #sql nvarchar(max)= '
bulk insert #Data
from ''C:\Users\Michał\Desktop\csvtest.csv''
with (
FIELDTERMINATOR = '',''
, ROWTERMINATOR = ''\n''
, lastrow = '+convert(nvarchar(11),#lastline)+'
);';
exec sp_executesql #sql;
select * from #Data
drop table #Data
If the lastrow option doesn't work for you, manually delete the last row and try to re-run your BulkLoad process. If everything works fine, crate a VB.NET executable to open the file, strip out the last row, and re-save the file. can schedule the delete-last-row process and the BulkLoad process using the Windows Task Scheduler.
https://www.sevenforums.com/tutorials/11949-elevated-program-shortcut-without-uac-prompt-create.html
I'm trying to import a .txt file into Advanced Query Tool (the SQL client I use). So far, I have:
CREATE TABLE #tb_test
(
id INTEGER,
name varchar(10),
dob date,
city char(20),
state char(20),
zip integer
);
insert into #tb_test
values
(1,'TEST','2015-01-01','TEST','TEST',11111)
;
bulk insert #tb_test
from 'h:\tbdata.txt'
with
(
fieldterminator = '\t',
rowterminator = '\n'
);
I receive an error message saying there's a syntax error on line 1. Am I missing a database from which #tb_test comes (like db.#tb_test)?
Here's a line from the tbdata.txt file:
2,'TEST2','2012-01-01','TEST','TEST',21111
I was curious with this question and I found the following solution:
Your data is comma separated but you are trying to split by TAB
two options: change the file data to be TAB separated or change the fieldterminator = '\t' to fieldterminator = ','
The DATE format has issues when loading directly from a file, my best solution is to change the temp field dob to type VARCHAR(20) and then, when passing to the final display/data storage convert to DATE.
Here is the corrected code:
CREATE TABLE #tb_test
(
id INTEGER,
name varchar(10),
dob varchar(20),
city char(20),
state char(20),
zip integer
);
insert into #tb_test
values
(1,'TEST','2015-01-01','TEST','TEST',11111)
;
bulk insert #tb_test
from 'h:\tbdata.txt'
with
(
fieldterminator = ',',
rowterminator = '\n'
);
Basically, I want to import hundreds of CSV files into SQL Server 2008.
File format is as following :
<Ticker>,<DTYYYYMMDD>,<Open>,<High>,<Low>,<Close>,<Volume>
AAM,20120110,21.6,22.8,21.4,21.6,3510
AAM,20120109,22.2,22.9,22.0,22.2,1130
AAM,20120105,0.0,23.0,22.2,22.2,210
I tried :
BULK
INSERT BBB
FROM 'D:\FIFA\excel_aam.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '/n'
)
GO
but it didn't work. So I was thinking import the CSV file as varchar format, then change each columns to proper data type later, like this :
CREATE TABLE BBB (
TICKER VARCHAR(15)NULL,
INDEXDATE VARCHAR(15) PRIMARY KEY,
OPENPRICE VARCHAR(15) NULL,
HIGHPRICE VARCHAR(15) NULL,
LOWPRICE VARCHAR(15) NULL,
CLOSEPRICE VARCHAR(15) NOT NULL,
VOLUME VARCHAR(15))
GO
but it gave me the error :
Msg 4863, Level 16, State 1, Line 1
Bulk load data conversion error (truncation) for row 1, column 7 (VOLUME).
So, how could I import these files (so many files that i couldn't use import and export wizard) into SQL Server properly?
For importing so many files sounds like u'll need SSIS
It works just fine in my case when I just change the rowterminator to \n (not /n)
--CREATE TABLE BBB (
--TICKER VARCHAR(15)NULL,
--INDEXDATE DATETIME,
--OPENPRICE DECIMAL(12,4),
--HIGHPRICE DECIMAL(12,4),
--LOWPRICE DECIMAL(12,4),
--CLOSEPRICE DECIMAL(12,4),
--VOLUME DECIMAL(20,4))
--GO
BULK INSERT BBB
FROM 'D:\FIFA\excel_aam.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
(3 row(s) affected)
and I have the rows in the BBB table now....
I have several chunks of a table in .dat files
I want to import all of these chuncks to a single sql server table.
To do it with one I do
BULK INSERT dbo.Tab
FROM 'C:\Data\1.dat'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
);
but how to append to the table the rest of the .dat files?
You fire multiple BULK INSERT commands.
BULK INSERT dbo.Tab
FROM 'C:\Data\1.dat'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
);
BULK INSERT dbo.Tab
FROM 'C:\Data\2.dat'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
);
...
Alternatively (and probably better for performance), use some other program to merge the files together first.