I am trying to export of table in csv file by using bcp command in microsoft sql server.
Below is the table sample data
Table name : XYZ
col1 col2 col3
abcd,inc. USD,inc 1234
pqrs,inc USD,inc 6789
stuv,inc USD,inc 0009
There is comma in column values as above.
I have written .fmt file like below:
test.fmt
13.0
3
1 SQLCHAR 0 4000 "\",\"" 1 col1 SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 0 4000 "\",\"" 2 col2 SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 0 4000 "\r\n" 3 col3 SQL_Latin1_General_CP1_CI_AS
Below is command I am using:
DECLARE
#V_BCP_QUERY VARCHAR(4000),
#V_BCP_OUTPUT_FILE VARCHAR(1500),
#V_BCP_FORMAT_FILE VARCHAR(1500),
#V_BCP_COMMAND VARCHAR(4000)
begin
SET #V_BCP_QUERY='"SELECT col1,col2,col3 FROM TABS..XYZ"'
SET #V_BCP_OUTPUT_FILE='"D:\OUTPUT.csv"'
SET #V_BCP_FORMAT_FILE='"D:\test.fmt"'
SET #V_BCP_COMMAND='bcp '+#V_BCP_QUERY+' queryout '+#V_BCP_OUTPUT_FILE+' -f '+#V_BCP_FORMAT_FILE+' -T -S "DEV-CR"'
EXECUTE Master.dbo.xp_CmdShell #V_BCP_COMMAND
end
I am getting below data in OUTPUT.csv file:
abcd,inc.","USD,inc","1234
pqrs,inc","USD,inc","6789
stuv,inc","USD,inc","0009
there is no " at start of line and end of line.
Also when I open this in excel then all rows are coming in a single line
my requirement is to export file in csv file.
Kindly help
You could hack a solution together - but using the correct tool for the job would be much easier and better in the long run.
Instead of using BCP to output individual columns, create a single column formatted with your desired result:
SELECT quotename(concat_ws(',', quotename(col1, char(34)), quotename(col2, char(34)), quotename(col3, char(34)), char(34))
FROM yourTable
This will give you a single column in your output - with double-quotes around the whole string, double-quotes around each column, concatenated with the '.' separator.
Sure it's ugly, but it's simple, quick and gets it done.
SELECT '"' + col1 + '",' +
'"' + col2 + '",' +
'"' + col3 + '"'
FROM Table
Related
I'm trying to load a file using bulk insert, but data isn't inserted correctly because some of the data is covered by quotation marks.
I've tried using a format file, but it doesn't work becasue not ALL the rows in that column contain quotes. Only some do. e.g.
columna
abc
cdf
"dfd"
dfs
"aee"
So my format file doesn't work.
My bulk insert code:
bulk insert tablename
from 'C:/...'
with
(
FIRSTROW = 2,
rowterminator = '0x0a'
,formatfile = 'file.fmt'
)
Format file:
10.0
5
1 SQLCHAR 0 1000 "," 1 "a" ""
2 SQLCHAR 0 1000 ",\"" 2 "b" ""
3 SQLCHAR 0 1000 "\",\"" 3 "d" <- has quotes ""
4 SQLCHAR 0 1000 ",\"" 4 "e" ""
5 SQLCHAR 0 1000 "\n" 5 "f"
Any ideas?
If there is no other way to remove double quotes in the column/s then what you can do is to do a post process where you update the affected column/s with replace
ie.
update mytable set col1 = replace(col1,'"',''),col2 = replace(col2,'"','')
I came across this problem with some data which also had some quotation marks inside the data so I couldn't use replace. Just in case there is this:
CREATE TABLE SomeTable
(
ColumnA VARCHAR(5)
)
INSERT INTO SomeTable VALUES ('abc')
INSERT INTO SomeTable VALUES ('cdf')
INSERT INTO SomeTable VALUES ('"dfd"')
INSERT INTO SomeTable VALUES ('dfs')
INSERT INTO SomeTable VALUES ('"aee"')
INSERT INTO SomeTable VALUES (' efg ')
GO
SELECT * FROM SomeTable
GO
--TRIM THE DATA
UPDATE SomeTable SET ColumnA =LTRIM(RTRIM(ColumnA))
GO
--DELETE THE DELIMITERS
UPDATE SomeTable SET ColumnA = LEFT(ColumnA,LEN(ColumnA)-1) WHERE RIGHT(ColumnA,1) LIKE '"'
GO
UPDATE SomeTable SET ColumnA = RIGHT(ColumnA,LEN(ColumnA)-1) WHERE LEFT(ColumnA,1) LIKE '"'
GO
--RETRIM THE DATA
UPDATE SomeTable SET ColumnA =LTRIM(RTRIM(ColumnA))
GO
SELECT * FROM SomeTable
GO
I'm using method below inserting Data from csv file into SQL.
BULK
INSERT tblMember
FROM 'F:\target.txt'
WITH
(
DATAFILETYPE='widechar',
CODEPAGE = 'ACP',
FIELDTERMINATOR = ';',
ROWTERMINATOR = '\n',
ERRORFILE = 'C:\CSVDATA\ErrorRows.csv',
)
GO
I need to do two things. First check if All Chars in Column One of CSV file for each row are only Digit and if yes Insert it. and Also I need to add a specific Word before these chars while inserting.
01 - 123,M,A,USA
02 - H24,N,Z,USA
I need to only insert row one, Because Column One is only Digit numbers '123', and I need to add "D" before this numbers and then insert it into SQL. so we have something like this is SQL after insertion:
"D123","M","A","USA"
Possible?
Lets consider a sample CSV(in C Drive) file target-c.txt which contain four lines of data.(Notice i have use target-c.txt not target.txt)
123,M,A,USA
H24,N,Z,USA
H25,N,V,USA
456,M,U,USA
Now create a Non-XML Format File(in C Drive) named it targetFormat.fmt. and populate the file in following way
9.0
4
1 SQLCHAR 0 100 "," 1 Col1 SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 0 100 "," 2 Col2 SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 0 100 "," 3 Col3 ""
4 SQLCHAR 0 100 "\r\n" 4 Col4 SQL_Latin1_General_CP1_CI_AS
Please Be Careful with this formatting.Click this Link if you want to read more about Non-XML Format File.The basic example would be like this.
Please change the format file according to your need.(like DataType, ChaterLength etc.)
I have created a sample table tblMember (please change according to your way, like column name , datatype etc. Remember you have to change the targetFormat.fmt file too)
CREATE TABLE tblMember
(
Col1 nvarchar(50),
Col2 nvarchar(50) ,
Col3 nvarchar(50) ,
Col4 nvarchar(50)
);
Then Use the following query for bulk-insert according to your way(its add a character "D" in front of Col1 with integer value)
INSERT INTO tblMember(Col1,Col2,Col3,Col4)
(
select 'D'+t1.Col1 AS Col1,t1.Col2,t1.Col3,t1.Col4
from openrowset(bulk 'C:\target-c.txt'
, formatfile = 'C:\targetFormat.fmt'
, firstrow = 1) as t1
where t1.Col1 not like '%[^0-9]%' --Not Like Letter Number mixed (123, 456)
UNION
select t1.Col1,t1.Col2,t1.Col3,t1.Col4
from openrowset(bulk 'C:\target-c.txt'
, formatfile = 'C:\targetFormat.fmt'
, firstrow = 1) as t1
where t1.Col1 like '%[^0-9]%'--Like Letter Number mixed (H24, H25)
)
Now if you select your table you will get this (i have tried and its working fine)
Here is your answer You can ordered the column if you want to. its very easy, just hold the query in a first bracket and order it or format it according your way.
I want to extract the CSV file from SQL Server and I manage to do that with the following code:
DECLARE #bcpCommand1 VARCHAR(2000)
SET #bcpCommand1 = 'bcp " SELECT * from ##tblUser " queryout'
SET #bcpCommand1 = #bcpCommand1 + ' C:\OutPut\CSV\My_Output.csv -c -w -T -S197.1.1.1 -Umyuser -Pmypwd","-CRAW'
EXEC CPS..xp_cmdshell #bcpCommand1
But the problem is ...
The big integer values are automatically coverted to scientific notation in CSV. Instead of this I want to display all the digits as it is.
I want to display the date as '01-Dec-2015 11:20 AM'.
If any data contains a comma, then the data is split to the next column. For example name is "mohan,Raj" then in the .csv data Mohan is in one column and raj in another column. How to avoid that?
Is there any way extract .csv using format file. How to do that?
Is there any way to do that? Please help me on this and post the code for that.
The BCP is used to copy whatever from a table to csv or txt or xml format.
If you want to get the comma separated values in one column as multiple column in csv. Then you have to modify your temp table accordingly.
I want to display the date as '01-Dec-2015 11:20 AM'.
There is not direct conversion to your exact format
you can use this code to do the same
SELECT REPLACE(CONVERT(VARCHAR(11), GETDATE(), 106), ' ', '-') +' '+STUFF(RIGHT( CONVERT(VARCHAR,GETDATE(),100 ) ,7), 6, 0, ' ')
4) Is there any way extract csv using format file.How to do that?
sqlcmd -S MyServer -d myDB -E -Q "select col1, col2, col3 from SomeTable"
-o "MyData.csv" -h-1 -s"," -w 700
-h-1 removes column name headers from the result
-s"," sets the column seperator to ,
-w 700 sets the row width to 700 chars
(this will need to be as wide as the longest row or it will wrap to the next line)
The answer on your 4th question (and it helps with questions 1 and 2). To load csv-data on server you can use:
INSERT INTO dbo.YourTable
SELECT a.* FROM OPENROWSET( BULK 'D:\our.csv', FORMATFILE = 'D:\our.fmt') AS a;
The sample of our.fmt (it's file that describes the fields in csv)
9.0
4
1 SQLCHAR 0 50 ";" 1 Field1 SQL_Latin1_General_Cp437_BIN
2 SQLCHAR 0 50 ";" 2 Field2 SQL_Latin1_General_Cp437_BIN
3 SQLCHAR 0 50 ";" 3 Field3 SQL_Latin1_General_Cp437_BIN
4 SQLCHAR 0 500 "\r\n" 4 Field4 SQL_Latin1_General_Cp437_BIN
You can find description of *.fmt here.
I have a field in my table which has multiple reason codes concatenated in 1 column.
e.g. 2 records
Reason_Codes
Record1: 001,002,004,009,010
Record2: 001,003,005,006
In my SSRS report the user will be searching for data using one of the above reason codes. e.g.
001 will retrieve both records.
005 will retrieve the second record
and so on.
Kindly advise how this can be achieved using SQL or Stored Procedure.
Many thanks.
If you are just passing in a single Reason Code to search on, you don't even need to bother with splitting the comma-separated list: you can just use a LIKE clause as follows:
SELECT tb.field1, tb.field2
FROM SchemaName.TableName tb
WHERE ',' + tb.Reason_Codes + ',' LIKE '%,' + #ReasonCode + ',%';
Try the following to see:
DECLARE #Bob TABLE (ID INT IDENTITY(1, 1) NOT NULL, ReasonCodes VARCHAR(50));
INSERT INTO #Bob (ReasonCodes) VALUES ('001,002,004,009,010');
INSERT INTO #Bob (ReasonCodes) VALUES ('001,003,005,006');
DECLARE #ReasonCode VARCHAR(5);
SET #ReasonCode = '001';
SELECT tb.ID, tb.ReasonCodes
FROM #Bob tb
WHERE ',' + tb.ReasonCodes + ',' LIKE '%,' + #ReasonCode + ',%';
-- returns both rows
SET #ReasonCode = '005';
SELECT tb.ID, tb.ReasonCodes
FROM #Bob tb
WHERE ',' + tb.ReasonCodes + ',' LIKE '%,' + #ReasonCode + ',%';
-- returns only row #2
I have blogged about something like this a long time ago. May be this will help: http://dotnetinternal.blogspot.com/2013/10/comma-separated-to-temp-table.html
The core solution would be to convert the comma separated values into a temporary table and then do a simple query on the temporary table to get your desired result.
I have downloaded the GeoLiteCountry CSV file from Maxmind - http://www.maxmind.com/app/geolitecountry. Using the format given to me as standard (so that this can become an automated task) I am attempting import all the data into a table.
I created a new table IPCountries2 which has columns exactly matching the columns provided:
FromIP varchar(50),
ToIP varchar(50),
BeginNum bigint,
EndNum bigint,
CountryCode varchar(50),
CountryName varchar(250)
Using the various chunks of code I could find, I was unable to get it working using the field terminator and row terminator:
BULK
INSERT CSVTest
FROM 'c:\csvtest.txt'
WITH
(
FIELDTERMINATOR = '","',
ROWTERMINATOR = '\n'
)
GO
The result of this was a single row inserted, all correct except the last one had overflowed with the next lines (presumably the whole database if I didn't have a limit). Also, the first cell had a quote at the start.
I looked around and found something called a format file (never used these). Made one which looks like:
10.0
6
1 SQLCHAR 0 50 "," 1 FromIP ""
2 SQLCHAR 0 50 "," 2 ToIP ""
3 SQLBIGINT 0 19 "," 3 BeginNum ""
4 SQLBIGINT 0 19 "," 4 EndNum ""
5 SQLCHAR 0 50 "," 5 CountryCode ""
6 SQLCHAR 0 250 "\n" 6 CountryName ""
but this errors on the bigint lines:
Msg 4867, Level 16, State 1, Line 1
Bulk load data conversion error (overflow) for row 1, column 3 (BeginNum).
It does that 10 times and then stops because of maximum error count.
I was able to get the first method working if I took it into Excel and re-saved, this removed the quotes. However, I don't want to rely on this method as I want this to update automatically every week and not have to open and re-save manually.
I don't mind which of the two methods I use ultimately, just so long as it works with a clean file. I had a look at their documentation but they only have code for PHP or MS Access.
Edit
Some lines from the CSV file:
"1.0.0.0","1.0.0.255","16777216","16777471","AU","Australia"
"1.0.1.0","1.0.3.255","16777472","16778239","CN","China"
"1.0.4.0","1.0.7.255","16778240","16779263","AU","Australia"
"1.0.8.0","1.0.15.255","16779264","16781311","CN","China"
"1.0.16.0","1.0.31.255","16781312","16785407","JP","Japan"
"1.0.32.0","1.0.63.255","16785408","16793599","CN","China"
"1.0.64.0","1.0.127.255","16793600","16809983","JP","Japan"
"1.0.128.0","1.0.255.255","16809984","16842751","TH","Thailand"
"1.1.0.0","1.1.0.255","16842752","16843007","CN","China"
"1.1.1.0","1.1.1.255","16843008","16843263","AU","Australia"
"1.1.2.0","1.1.63.255","16843264","16859135","CN","China"
"1.1.64.0","1.1.127.255","16859136","16875519","JP","Japan"
"1.1.128.0","1.1.255.255","16875520","16908287","TH","Thailand"
Update
After some persisting I was able to get things working 95% with the original method (without the format document). However, it was changed slightly to look like so:
BULK INSERT IPCountries2
FROM 'c:\Temp\GeoIPCountryWhois.csv'
WITH
(
FIELDTERMINATOR = '","',
ROWTERMINATOR = '"'
)
GO
Everything goes in the right fields as they should, the only issue I have is in the first column there is a quote at the beginning. Some sample data:
FromIP ToIP BeginNum EndNum CountryCode Country
"2.21.248.0 2.21.253.255 34994176 34995711 FR France
"2.21.254.0 2.21.254.255 34995712 34995967 EU Europe
"2.21.255.0 2.21.255.255 34995968 34996223 NL Netherlands
Success. Searching around and some help from another forum finally got me to my solution. For those in need of a similar solution, keep reading:
I ended up using the format file method - whether it would be possible to use fieldterminators and row terminators I'm not sure.
My SQL code looks like:
CREATE TABLE #TempTable
(
DuffColumn varchar(50),
FromIP varchar(50),
ToIP varchar(50),
BeginNum bigint,
EndNum bigint,
CountryCode varchar(50),
CountryName varchar(250)
)
BULK
INSERT #TempTable
FROM 'c:\Temp\GeoIPCountryWhois.csv'
WITH
(
FORMATFILE = 'C:\Temp\format.fmt'
)
INSERT INTO IPCountries2 (FromIP, ToIP, BeginNum, EndNum, CountryCode, Country)
SELECT FromIP, ToIP, BeginNum, EndNum, CountryCode, CountryName FROM #TempTable
As found in my research, it was necessary to have a useless column which simply captured the first quote.
My format file looks like:
10.0
7
1 SQLCHAR 0 1 "" 1 DuffColumn ""
2 SQLCHAR 0 50 "\",\"" 2 FromIP ""
3 SQLCHAR 0 50 "\",\"" 3 ToIP ""
4 SQLCHAR 0 19 "\",\"" 4 BeginNum ""
5 SQLCHAR 0 19 "\",\"" 5 EndNum ""
6 SQLCHAR 0 50 "\",\"" 6 CountryCode ""
7 SQLCHAR 0 250 "\"\n" 7 CountryName ""
To note, despite eventually being stored as a BIGINT, BeginNum and EndNum are both passed in as SQLCHARS, otherwise the insert does an odd multiplication on the numbers (something about reading it as bytes rather than digits, I didn't entirely understand it).
And that's about it. The last thing to automate this script fully is to truncate the table first so as to clear out old records. However that might not be to everyones needs.
Try this command. All I did is remove the double quotes from your FIELDTERMINATOR:
BULK
INSERT CSVTest
FROM 'c:\csvtest.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
Your data fields are actually terminated by commas, not commas wrapped in quotes. I also suggest building a staging/import table match the datatypes of your source file exactly, which in this case would look like:
FromIP varchar(50),
ToIP varchar(50),
BeginNum varchar(50),
EndNum varchar(50),
CountryCode varchar(50),
CountryName varchar(250)
Your source data for BeginNum and EndNum is actually string, not bigint. You can convert this data once you have it imported into your staging table.
declare #sql varchar(1000)
declare #filename varchar(100) = 'C:\Temp\GeoIPCountryWhois.csv'
set #sql =
'BULK INSERT geoip FROM ''' + #filename + '''
WITH
(
CHECK_CONSTRAINTS,
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''' + char(0x0A) + '''
)'
exec (#sql)