Basically, I want to import hundreds of CSV files into SQL Server 2008.
File format is as following :
<Ticker>,<DTYYYYMMDD>,<Open>,<High>,<Low>,<Close>,<Volume>
AAM,20120110,21.6,22.8,21.4,21.6,3510
AAM,20120109,22.2,22.9,22.0,22.2,1130
AAM,20120105,0.0,23.0,22.2,22.2,210
I tried :
BULK
INSERT BBB
FROM 'D:\FIFA\excel_aam.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '/n'
)
GO
but it didn't work. So I was thinking import the CSV file as varchar format, then change each columns to proper data type later, like this :
CREATE TABLE BBB (
TICKER VARCHAR(15)NULL,
INDEXDATE VARCHAR(15) PRIMARY KEY,
OPENPRICE VARCHAR(15) NULL,
HIGHPRICE VARCHAR(15) NULL,
LOWPRICE VARCHAR(15) NULL,
CLOSEPRICE VARCHAR(15) NOT NULL,
VOLUME VARCHAR(15))
GO
but it gave me the error :
Msg 4863, Level 16, State 1, Line 1
Bulk load data conversion error (truncation) for row 1, column 7 (VOLUME).
So, how could I import these files (so many files that i couldn't use import and export wizard) into SQL Server properly?
For importing so many files sounds like u'll need SSIS
It works just fine in my case when I just change the rowterminator to \n (not /n)
--CREATE TABLE BBB (
--TICKER VARCHAR(15)NULL,
--INDEXDATE DATETIME,
--OPENPRICE DECIMAL(12,4),
--HIGHPRICE DECIMAL(12,4),
--LOWPRICE DECIMAL(12,4),
--CLOSEPRICE DECIMAL(12,4),
--VOLUME DECIMAL(20,4))
--GO
BULK INSERT BBB
FROM 'D:\FIFA\excel_aam.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
(3 row(s) affected)
and I have the rows in the BBB table now....
Related
I'm trying to create a table in SQL Server from a text file using bulk insert but I keep getting bulk load data conversion error(truncation). Is there something I'm doing wrong? The top part is how the data is in the text and below is the code.
'01','INPATIENT FACILITY','010','ACUTE CARE HOSPITAL'
'01','INPATIENT FACILITY','011','PRIVATE PSYCHIATRIC HOSPITAL'
'01','INPATIENT FACILITY','012','INPATIENT MEDICAL REHAB HOSPITAL'
CREATE TABLE [dbo].[PROVIDER_TYPE]
(
[PROVIDER_TYPE_ID] [VARCHAR](2) NULL,
[PROVIDER_TYPE] [VARCHAR](50) NULL,
[PROVIDER_SPECIALITY_ID] [VARCHAR](3) NULL,
[PROVIDER_SPECIALITY] [VARCHAR](50) NULL
) ON [PRIMARY]
BULK INSERT DBO.PROVIDER_TYPE FROM 'C:\SQL\t2.txt'
WITH (
datafiletype = 'char'
,fieldterminator = ','
,ROWTERMINATOR = '\n'
)
The first value isn't 2 characters long, it's 4. The value is '01'; that's inclusive of the single quotes ('). This is why you're getting a truncation error, as '01' ('''01''' if you were to want to represent the string in T-SQL) doesn't fit in a varchar(2).
If you're on SQL Server 2017+ you can use the FORMAT and FIELDQUOTE options. Note that I also use \r\n for ROWTERMINATOR, as I had both when I created the file, if yours only contains a line break (and no carriage return), then just use \n:
BULK INSERT dbo.PROVIDER_TYPE FROM '/mnt/WDBlue/t2.txt' --This was my test file
WITH (DATAFILETYPE = 'char',
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\r\n',
FORMAT = 'CSV',
FIELDQUOTE = '''');
If you aren't using SQL Server 2017+, then it simply does not support quoted fields, and I suggest using a different tool.
I have one problem with bulk insert. I have about 7 million records in the csv file, when I make the following query, just half of the total records are entered in the table.
The Query I use:
BULK INSERT *Table_Name* FROM 'file path'
WITH
(
FIRSTROW = 1, -- csv dont have header
FIELDTERMINATOR = '0x0a',
ROWTERMINATOR = '\n',
ERRORFILE = '*error file path*',
TABLOCK
)
Table Creation query:
CREATE TABLE *Table_Name*
(
ID int NOT NULL IDENTITY(1,1) PRIMARY KEY,
*Column_Name* varchar(50)
);
Where I make a mistake?
How to load all records from a csv file (all 7 million)?
I want to bulk import from CSV file in sql but \n for new line is not working in SQL as row terminator. it does not read any record from csv file if i use \n but when i use
ROWTERMINATOR = '0x0A'
it mix up the all records.
this is code what i am using in my sp.
Create Table #temp
(
Field1 nvarchar(max) null,
Field2 nvarchar(max) null,
Field3 nvarchar(max) null
)
BULK INSERT #temp
FROM 'c:\file.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --not working
--ROWTERMINATOR = '\r', --not working
--ROWTERMINATOR = char(10), ---not working
--ROWTERMINATOR = char(13), ---not working
TABLOCK
)
INSERT INTO table_name
(
tbl_field1,tbl_field2,tbl_field3
)
SELECT
field1,
field2,
field3
FROM #temp
Thanks in Advance
I did it with the help of #DVO. Thank you #dvo for answer. It is working fine as per your instructions. i used notepad++ and see the hidden characters and handle them accordingly.
I need to read data from .csv file which contains many records, but in last row
there is string "the_end" and after that there is no LF sign.
Here is my csv file:
1,James,Smith,19750101
2,Meggie,Smith,19790122
3,Robert,Smith,20071101
4,Alex,Smith,20040202
the_end
Below my sql script which reads data into temporary table:
create table #Data
(
id int,
first_name varchar(50),
last_name varchar(50),
birthdate smalldatetime
)
bulk insert #Data
from 'C:\Users\Michał\Desktop\csvtest.csv'
with
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
select * from #Data
drop table #Data
Error is:
Msg 4832, Level 16, State 1, Line 13
Bulk load: An unexpected end of file was encountered in the data file.
Msg 7399, Level 16, State 1, Line 13
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 13
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".
I want to read all data into my table without this last line label "the_end", that's obvious. How can i manage that? I cannot modify this file, because it's given from outside and I cannot do that.
Use the lastrow option
bulk insert #Data
from 'C:\Users\Michał\Desktop\csvtest.csv'
with
(
FIELDTERMINATOR = ','
, ROWTERMINATOR = '\n'
, lastrow = 4
)
dynamic sql example using a variable for lastline:
create table #Data (
id int
, first_name varchar(50)
, last_name varchar(50)
, birthdate smalldatetime
);
declare #lastline int = 4;
declare #sql nvarchar(max)= '
bulk insert #Data
from ''C:\Users\Michał\Desktop\csvtest.csv''
with (
FIELDTERMINATOR = '',''
, ROWTERMINATOR = ''\n''
, lastrow = '+convert(nvarchar(11),#lastline)+'
);';
exec sp_executesql #sql;
select * from #Data
drop table #Data
If the lastrow option doesn't work for you, manually delete the last row and try to re-run your BulkLoad process. If everything works fine, crate a VB.NET executable to open the file, strip out the last row, and re-save the file. can schedule the delete-last-row process and the BulkLoad process using the Windows Task Scheduler.
https://www.sevenforums.com/tutorials/11949-elevated-program-shortcut-without-uac-prompt-create.html
I'm trying to import a .txt file into Advanced Query Tool (the SQL client I use). So far, I have:
CREATE TABLE #tb_test
(
id INTEGER,
name varchar(10),
dob date,
city char(20),
state char(20),
zip integer
);
insert into #tb_test
values
(1,'TEST','2015-01-01','TEST','TEST',11111)
;
bulk insert #tb_test
from 'h:\tbdata.txt'
with
(
fieldterminator = '\t',
rowterminator = '\n'
);
I receive an error message saying there's a syntax error on line 1. Am I missing a database from which #tb_test comes (like db.#tb_test)?
Here's a line from the tbdata.txt file:
2,'TEST2','2012-01-01','TEST','TEST',21111
I was curious with this question and I found the following solution:
Your data is comma separated but you are trying to split by TAB
two options: change the file data to be TAB separated or change the fieldterminator = '\t' to fieldterminator = ','
The DATE format has issues when loading directly from a file, my best solution is to change the temp field dob to type VARCHAR(20) and then, when passing to the final display/data storage convert to DATE.
Here is the corrected code:
CREATE TABLE #tb_test
(
id INTEGER,
name varchar(10),
dob varchar(20),
city char(20),
state char(20),
zip integer
);
insert into #tb_test
values
(1,'TEST','2015-01-01','TEST','TEST',11111)
;
bulk insert #tb_test
from 'h:\tbdata.txt'
with
(
fieldterminator = ',',
rowterminator = '\n'
);