I need to read data from .csv file which contains many records, but in last row
there is string "the_end" and after that there is no LF sign.
Here is my csv file:
1,James,Smith,19750101
2,Meggie,Smith,19790122
3,Robert,Smith,20071101
4,Alex,Smith,20040202
the_end
Below my sql script which reads data into temporary table:
create table #Data
(
id int,
first_name varchar(50),
last_name varchar(50),
birthdate smalldatetime
)
bulk insert #Data
from 'C:\Users\Michał\Desktop\csvtest.csv'
with
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
select * from #Data
drop table #Data
Error is:
Msg 4832, Level 16, State 1, Line 13
Bulk load: An unexpected end of file was encountered in the data file.
Msg 7399, Level 16, State 1, Line 13
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 13
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
I want to read all data into my table without this last line label "the_end", that's obvious. How can i manage that? I cannot modify this file, because it's given from outside and I cannot do that.
Use the lastrow option
bulk insert #Data
from 'C:\Users\Michał\Desktop\csvtest.csv'
with
(
FIELDTERMINATOR = ','
, ROWTERMINATOR = '\n'
, lastrow = 4
)
dynamic sql example using a variable for lastline:
create table #Data (
id int
, first_name varchar(50)
, last_name varchar(50)
, birthdate smalldatetime
);
declare #lastline int = 4;
declare #sql nvarchar(max)= '
bulk insert #Data
from ''C:\Users\Michał\Desktop\csvtest.csv''
with (
FIELDTERMINATOR = '',''
, ROWTERMINATOR = ''\n''
, lastrow = '+convert(nvarchar(11),#lastline)+'
);';
exec sp_executesql #sql;
select * from #Data
drop table #Data
If the lastrow option doesn't work for you, manually delete the last row and try to re-run your BulkLoad process. If everything works fine, crate a VB.NET executable to open the file, strip out the last row, and re-save the file. can schedule the delete-last-row process and the BulkLoad process using the Windows Task Scheduler.
https://www.sevenforums.com/tutorials/11949-elevated-program-shortcut-without-uac-prompt-create.html
Related
I have one problem with bulk insert. I have about 7 million records in the csv file, when I make the following query, just half of the total records are entered in the table.
The Query I use:
BULK INSERT *Table_Name* FROM 'file path'
WITH
(
FIRSTROW = 1, -- csv dont have header
FIELDTERMINATOR = '0x0a',
ROWTERMINATOR = '\n',
ERRORFILE = '*error file path*',
TABLOCK
)
Table Creation query:
CREATE TABLE *Table_Name*
(
ID int NOT NULL IDENTITY(1,1) PRIMARY KEY,
*Column_Name* varchar(50)
);
Where I make a mistake?
How to load all records from a csv file (all 7 million)?
I'm trying to load a CSV file into a table in order to sort the data. However, the smallmoney column BillingRate (e.g. $203.75) will not convert, with SQL Server producing the following message:
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 2, column 4 (BillingRate).
Msg 4864, Level 16, State 1, Line 10
Here is the code I am using in order to do this:
--CREATE TABLE SubData2
--(
--RecordID int,
--SubscriberID int,
--BillingMonth int,
--BillingRate smallmoney,
--Region varchar(255)
--);
BULK INSERT Subdata2
FROM 'C:\Folder\1-caseint.csv'
WITH
(FIRSTROW = 2,
FIELDTERMINATOR = '|', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'C:\Folder\CaseErrorRows4.csv',
TABLOCK);
A typical line of code in the CSV files looks like:
1|0000000001|1|$233.94|"West"
Apologies if there are any glaring errors here - I'm new to SQL Server :)
Many thanks in advance,
Tom.
This is odd. On a direct insert only the last fails.
declare #t table (sm smallmoney);
insert into #t
values ('$256.6')
insert into #t
values (244.8);
insert into #t
values ('12.8');
insert into #t
values ('$256.5'), ('244.7');
insert into #t
values ($256.5);
insert into #t
values ('$256.5'), (244.7), ('12.12');
select * from #t;
Give it a try removing the $ from the data.
I have an excel file that I want to bulk insert into temp table:
create table #tmptable
(
Date varchar(10),
Receipt varchar(50),
Description varchar(100),
[Card Member] varchar(50),
[Account #] varchar(17),
Amount varchar(20)
)
bulk insert #tmptable
from 'C:\Transactions\example.xls'
with (FieldTerminator='\t', RowTerminator = '\n')
go
This is my excel file:
When executing bulk statement, getting the following error:
Msg 4863, Level 16, State 1, Line 1 Bulk load data conversion error
(truncation) for row 1, column 1 (Date). Msg 4864, Level 16, State 1,
Line 1 Bulk load data conversion error (type mismatch or invalid
character for the specified codepage) for row 2, column 1 (Date).
Do not know why it happens.
Well, you are actually reading your headers, meaning the the data on the first few rows of your xls are images that's why you are getting a type mismatch error
get the row number of that first row where the data actually is.
then you use this:
create table #tmptable
(
Date date,
Receipt varchar(50),
Description varchar(100),
[Card Member] varchar(50),
[Account #] varchar(17),
Amount varchar(20)
)
bulk insert #tmptable
from 'C:\Transactions\example.xls'
with (FieldTerminator='\t', RowTerminator = '\n', FirstRow = X)
go
where X is the row number where the data actually starts and not the headers
I'm trying to import a .txt file into Advanced Query Tool (the SQL client I use). So far, I have:
CREATE TABLE #tb_test
(
id INTEGER,
name varchar(10),
dob date,
city char(20),
state char(20),
zip integer
);
insert into #tb_test
values
(1,'TEST','2015-01-01','TEST','TEST',11111)
;
bulk insert #tb_test
from 'h:\tbdata.txt'
with
(
fieldterminator = '\t',
rowterminator = '\n'
);
I receive an error message saying there's a syntax error on line 1. Am I missing a database from which #tb_test comes (like db.#tb_test)?
Here's a line from the tbdata.txt file:
2,'TEST2','2012-01-01','TEST','TEST',21111
I was curious with this question and I found the following solution:
Your data is comma separated but you are trying to split by TAB
two options: change the file data to be TAB separated or change the fieldterminator = '\t' to fieldterminator = ','
The DATE format has issues when loading directly from a file, my best solution is to change the temp field dob to type VARCHAR(20) and then, when passing to the final display/data storage convert to DATE.
Here is the corrected code:
CREATE TABLE #tb_test
(
id INTEGER,
name varchar(10),
dob varchar(20),
city char(20),
state char(20),
zip integer
);
insert into #tb_test
values
(1,'TEST','2015-01-01','TEST','TEST',11111)
;
bulk insert #tb_test
from 'h:\tbdata.txt'
with
(
fieldterminator = ',',
rowterminator = '\n'
);
Basically, I want to import hundreds of CSV files into SQL Server 2008.
File format is as following :
<Ticker>,<DTYYYYMMDD>,<Open>,<High>,<Low>,<Close>,<Volume>
AAM,20120110,21.6,22.8,21.4,21.6,3510
AAM,20120109,22.2,22.9,22.0,22.2,1130
AAM,20120105,0.0,23.0,22.2,22.2,210
I tried :
BULK
INSERT BBB
FROM 'D:\FIFA\excel_aam.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '/n'
)
GO
but it didn't work. So I was thinking import the CSV file as varchar format, then change each columns to proper data type later, like this :
CREATE TABLE BBB (
TICKER VARCHAR(15)NULL,
INDEXDATE VARCHAR(15) PRIMARY KEY,
OPENPRICE VARCHAR(15) NULL,
HIGHPRICE VARCHAR(15) NULL,
LOWPRICE VARCHAR(15) NULL,
CLOSEPRICE VARCHAR(15) NOT NULL,
VOLUME VARCHAR(15))
GO
but it gave me the error :
Msg 4863, Level 16, State 1, Line 1
Bulk load data conversion error (truncation) for row 1, column 7 (VOLUME).
So, how could I import these files (so many files that i couldn't use import and export wizard) into SQL Server properly?
For importing so many files sounds like u'll need SSIS
It works just fine in my case when I just change the rowterminator to \n (not /n)
--CREATE TABLE BBB (
--TICKER VARCHAR(15)NULL,
--INDEXDATE DATETIME,
--OPENPRICE DECIMAL(12,4),
--HIGHPRICE DECIMAL(12,4),
--LOWPRICE DECIMAL(12,4),
--CLOSEPRICE DECIMAL(12,4),
--VOLUME DECIMAL(20,4))
--GO
BULK INSERT BBB
FROM 'D:\FIFA\excel_aam.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
(3 row(s) affected)
and I have the rows in the BBB table now....