Related
I am trying to export of table in csv file by using bcp command in microsoft sql server.
Below is the table sample data
Table name : XYZ
col1 col2 col3
abcd,inc. USD,inc 1234
pqrs,inc USD,inc 6789
stuv,inc USD,inc 0009
There is comma in column values as above.
I have written .fmt file like below:
test.fmt
13.0
3
1 SQLCHAR 0 4000 "\",\"" 1 col1 SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 0 4000 "\",\"" 2 col2 SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 0 4000 "\r\n" 3 col3 SQL_Latin1_General_CP1_CI_AS
Below is command I am using:
DECLARE
#V_BCP_QUERY VARCHAR(4000),
#V_BCP_OUTPUT_FILE VARCHAR(1500),
#V_BCP_FORMAT_FILE VARCHAR(1500),
#V_BCP_COMMAND VARCHAR(4000)
begin
SET #V_BCP_QUERY='"SELECT col1,col2,col3 FROM TABS..XYZ"'
SET #V_BCP_OUTPUT_FILE='"D:\OUTPUT.csv"'
SET #V_BCP_FORMAT_FILE='"D:\test.fmt"'
SET #V_BCP_COMMAND='bcp '+#V_BCP_QUERY+' queryout '+#V_BCP_OUTPUT_FILE+' -f '+#V_BCP_FORMAT_FILE+' -T -S "DEV-CR"'
EXECUTE Master.dbo.xp_CmdShell #V_BCP_COMMAND
end
I am getting below data in OUTPUT.csv file:
abcd,inc.","USD,inc","1234
pqrs,inc","USD,inc","6789
stuv,inc","USD,inc","0009
there is no " at start of line and end of line.
Also when I open this in excel then all rows are coming in a single line
my requirement is to export file in csv file.
Kindly help
You could hack a solution together - but using the correct tool for the job would be much easier and better in the long run.
Instead of using BCP to output individual columns, create a single column formatted with your desired result:
SELECT quotename(concat_ws(',', quotename(col1, char(34)), quotename(col2, char(34)), quotename(col3, char(34)), char(34))
FROM yourTable
This will give you a single column in your output - with double-quotes around the whole string, double-quotes around each column, concatenated with the '.' separator.
Sure it's ugly, but it's simple, quick and gets it done.
SELECT '"' + col1 + '",' +
'"' + col2 + '",' +
'"' + col3 + '"'
FROM Table
I'm trying to load a file using bulk insert, but data isn't inserted correctly because some of the data is covered by quotation marks.
I've tried using a format file, but it doesn't work becasue not ALL the rows in that column contain quotes. Only some do. e.g.
columna
abc
cdf
"dfd"
dfs
"aee"
So my format file doesn't work.
My bulk insert code:
bulk insert tablename
from 'C:/...'
with
(
FIRSTROW = 2,
rowterminator = '0x0a'
,formatfile = 'file.fmt'
)
Format file:
10.0
5
1 SQLCHAR 0 1000 "," 1 "a" ""
2 SQLCHAR 0 1000 ",\"" 2 "b" ""
3 SQLCHAR 0 1000 "\",\"" 3 "d" <- has quotes ""
4 SQLCHAR 0 1000 ",\"" 4 "e" ""
5 SQLCHAR 0 1000 "\n" 5 "f"
Any ideas?
If there is no other way to remove double quotes in the column/s then what you can do is to do a post process where you update the affected column/s with replace
ie.
update mytable set col1 = replace(col1,'"',''),col2 = replace(col2,'"','')
I came across this problem with some data which also had some quotation marks inside the data so I couldn't use replace. Just in case there is this:
CREATE TABLE SomeTable
(
ColumnA VARCHAR(5)
)
INSERT INTO SomeTable VALUES ('abc')
INSERT INTO SomeTable VALUES ('cdf')
INSERT INTO SomeTable VALUES ('"dfd"')
INSERT INTO SomeTable VALUES ('dfs')
INSERT INTO SomeTable VALUES ('"aee"')
INSERT INTO SomeTable VALUES (' efg ')
GO
SELECT * FROM SomeTable
GO
--TRIM THE DATA
UPDATE SomeTable SET ColumnA =LTRIM(RTRIM(ColumnA))
GO
--DELETE THE DELIMITERS
UPDATE SomeTable SET ColumnA = LEFT(ColumnA,LEN(ColumnA)-1) WHERE RIGHT(ColumnA,1) LIKE '"'
GO
UPDATE SomeTable SET ColumnA = RIGHT(ColumnA,LEN(ColumnA)-1) WHERE LEFT(ColumnA,1) LIKE '"'
GO
--RETRIM THE DATA
UPDATE SomeTable SET ColumnA =LTRIM(RTRIM(ColumnA))
GO
SELECT * FROM SomeTable
GO
I'm using method below inserting Data from csv file into SQL.
BULK
INSERT tblMember
FROM 'F:\target.txt'
WITH
(
DATAFILETYPE='widechar',
CODEPAGE = 'ACP',
FIELDTERMINATOR = ';',
ROWTERMINATOR = '\n',
ERRORFILE = 'C:\CSVDATA\ErrorRows.csv',
)
GO
I need to do two things. First check if All Chars in Column One of CSV file for each row are only Digit and if yes Insert it. and Also I need to add a specific Word before these chars while inserting.
01 - 123,M,A,USA
02 - H24,N,Z,USA
I need to only insert row one, Because Column One is only Digit numbers '123', and I need to add "D" before this numbers and then insert it into SQL. so we have something like this is SQL after insertion:
"D123","M","A","USA"
Possible?
Lets consider a sample CSV(in C Drive) file target-c.txt which contain four lines of data.(Notice i have use target-c.txt not target.txt)
123,M,A,USA
H24,N,Z,USA
H25,N,V,USA
456,M,U,USA
Now create a Non-XML Format File(in C Drive) named it targetFormat.fmt. and populate the file in following way
9.0
4
1 SQLCHAR 0 100 "," 1 Col1 SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 0 100 "," 2 Col2 SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 0 100 "," 3 Col3 ""
4 SQLCHAR 0 100 "\r\n" 4 Col4 SQL_Latin1_General_CP1_CI_AS
Please Be Careful with this formatting.Click this Link if you want to read more about Non-XML Format File.The basic example would be like this.
Please change the format file according to your need.(like DataType, ChaterLength etc.)
I have created a sample table tblMember (please change according to your way, like column name , datatype etc. Remember you have to change the targetFormat.fmt file too)
CREATE TABLE tblMember
(
Col1 nvarchar(50),
Col2 nvarchar(50) ,
Col3 nvarchar(50) ,
Col4 nvarchar(50)
);
Then Use the following query for bulk-insert according to your way(its add a character "D" in front of Col1 with integer value)
INSERT INTO tblMember(Col1,Col2,Col3,Col4)
(
select 'D'+t1.Col1 AS Col1,t1.Col2,t1.Col3,t1.Col4
from openrowset(bulk 'C:\target-c.txt'
, formatfile = 'C:\targetFormat.fmt'
, firstrow = 1) as t1
where t1.Col1 not like '%[^0-9]%' --Not Like Letter Number mixed (123, 456)
UNION
select t1.Col1,t1.Col2,t1.Col3,t1.Col4
from openrowset(bulk 'C:\target-c.txt'
, formatfile = 'C:\targetFormat.fmt'
, firstrow = 1) as t1
where t1.Col1 like '%[^0-9]%'--Like Letter Number mixed (H24, H25)
)
Now if you select your table you will get this (i have tried and its working fine)
Here is your answer You can ordered the column if you want to. its very easy, just hold the query in a first bracket and order it or format it according your way.
I have a basic Select query which brings me back a set of results (roughly around 100,000 records) which I currently have to export to .CSV Format , issue is I then have to remove the commas out of the results and replace it with a (~). Which I do using a file format application that I got off the web.
But I'm trying to go about making this automated (if possible) to save time. Like e.g run off a stored procedure that can do this for me export the file in a (~) format.
Does anyone have any tips how this stored procedure can be written or an pointers would be appreciated.
p.s I have tried to use the export wizard but it just just crashes due to too many records.
Expected Result
Test1~Test2~Test3
5~6~7
(sql Script which I am running)
select
'SPK' as [AGENCY_CODE], -- should be set to SPK
'OBCALL' as [MEDIA_CODE], -- should be set to OBCALL
isnull(c.salutation,'') as [TITLE],
isnull(c.otherName,'') as [FORENAME],
isnull(c.name,'') as [SURNAME],
isnull(c.attTXT64,'') as [STANDARDISED_NAME],
replace(isnull(c.addr1, ''), ',', '.') AS [BEST_ADDRESS_LINE_1],
replace(isnull(c.addr2, ''), ',', '.') AS [BEST_ADDRESS_LINE_2],
replace(isnull(c.addr3, ''), ',', '.') AS [BEST_ADDRESS_LINE_3],
replace(isnull(c.addr4, ''), ',', '.') AS [BEST_ADDRESS_LINE_4],
isnull(c.postCode,'') as [BEST_POSTCODE],
--'0' + isnull(c.phone1,'') as [TELEPHONE_N2O],-- should be populated with the spare field ORIG_TEL .
RIGHT('0' + CONVERT(VARCHAR(11), c.phone1), 11) as [TELEPHONE_NO],-- should be populated with the spare field ORIG_TEL .
convert(varchar(100),c.attDT03,120) as [DATE_TIMESTAMP],
isnull(c.attTXT10,'') as [SM_CONTACT_KEY],
isnull(c.attTXT89,'') as [ SM_ADDRESS_KEY],
isnull(c.attTXT11,'') as [ CAMPAIGN_IDENTIFIER],
isnull(c.attTXT12,'') as [ WAVE_ID],
isnull(c.attTXT13,'') as [OLDSTACK_NEWSTACK_FLAG],
isnull(c.attTXT14,'') as [MARKET_3_FLAG],
isnull(c.attTXT15,'') as [ADSL_2_FLAG],
isnull(c.attTXT16,'') as [FIBRE_FLAG],
isnull(c.attTXT17,'') as [LOAD_ID],
isnull(c.attTXT18,'') as [CONTACT_POINT_KEY],
isnull(c.attTXT19,'') as [DATA_POOL_URN],
isnull(c.attTXT20,'') as [EVENT_KEY],
isnull(c.attTXT21,'') as [BILLING_ACCOUNT_KEY] ,
isnull(c.attTXT22,'') as [CAMPAIGN_SOURCE] ,
isnull(c.attTXT23,'') as [CAMPAIGN_CODE] ,
isnull(c.attTXT24,'') as [ CMT_ROLE_KEY],
isnull(c.attTXT25,'') as [ CMT_LOCATION_KEY],
isnull(c.attTXT26,'') as [BILL_ACCNT_NUM],
isnull(c.attTXT27,'') as [BILLING_ACCOUNT_TYPE],
--All other fields are as per the import record values
--DATE/TIME_OF_CONTRACT to CONTRACT_END_DATE_SUPPLIER3
CASE WHEN dx.datetime IS NULL THEN convert(varchar,getdate(),120) ELSE CONVERT(varchar, dx.datetime, 120) END
as [DATE/TIMEOF CONTACT],
Case when uc.campaignid = 3 then 'CT001' when uc.campaignid = 22 then 'CT001' when uc.campaignid = 18 then 'CT011'
when uc.campaignid = 26 then 'CT013'
end as [CAMPAIGN_TYPE], -- map to BT/DATA/10.CAMPAIGN_Code (return CAMPAIGN_TYPE)
ISNULL(( CASE
WHEN dx.[Abandon] = 1 THEN 'OC039'
ELSE d.code END),'OC042')
AS [OUTCOME_CODE], -- populate with (OC001-OCxxx)
'TM' as [CHANNEL_MEDIA_CODE], --= (TBC)
isnull(c.email,'') as [EMAIL_ADDRESS], -- populate with EMAIL_ADDRESS
'' as [EMAIL_CONSENT], --= populate with EMAIL_CONSENT ***************
'' as [INBOUND_TELEPHONE_NUMBER], --*****************
'' as [COMPETITOR_SUPPLIER_1], -- CONT CONTRACT_END_DATE_SUPPLIER3 – poplulate ***********
'' as [PRODUCT_FROM_SUPPLIER1], -- populate ************
'' as [CONTRACT_START_DATE_SUPPLIER1], --*****
'' as [CONTRACT_END_DATE_SUPPLIER1], --*****
'' as [COMPETITOR_SUPPLIER2], --****
'' as [PRODUCT_FROM_SUPPLIER2], --****
'' as [CONTRACT_START_DATE_SUPPLIER2],-- *****
'' as [CONTRACT_END_DATE_SUPPLIER2], --*****
'' as [COMPETITOR_SUPPLIER3],-- ****
'' as [PRODUCT_FROM_SUPPLIER3],--****
'' as [CONTRACT_START_DATE_SUPPLIER3],-- *****
'' as [CONTRACT_END_DATE_SUPPLIER3],-- *****
--ORDER_NUM to NUMBER_OF_CALLS_MADE
isnull(c.attTXT02,'') as [ORDER_NUM], -- Captured by Operator
isnull(dx.duration,0) as [CALL_DURATION],
-------------dxi.talk as [CALL_DURATION], -- populate difference start/end time (seconds)******
isnull(c.attTXT09,'') as [WARMTH_RATING_NOW], -- captured by agent
isnull(c.attTXT80,'') as [WARMTH_RATING_FUTURE_CAMPAIGNS], -- captured by agent
isnull(dx.callid,'') [SOURCE_INTERACTION_ID],
-------------isnull(a.id,'') as [SOURCE_INTERACTION_ID], -- Tpoints unique call Identifier --- Activity ID *******
isnull(uc.callcount,'') as [NUMBER_OF_CALLS_MADE],
(select left(ethnicOrigin, 1)) as [CALL_CONSENT_VALIDATE_FLG], -- ******** gift aid status - first byte only
(select left(nationality, 5)) as [CALL_CONSENT_OUTCOME_CD], -- ******** nationality - first 5 bytes only
--QUESTION _1 to ANSWER_3
isnull(c.attTXT56,'') as [QUESTION_1], -- populate from SPARE_FIELD_28
isnull(c.attTXT51,'') as [ANSWER_1], -- captured by the agent if SPARE_FIELD_1 populated
isnull(c.attTXT29,'') as [QUESTION_2], -- poaddingpulate from SPARE_FIELD_28
isnull(c.attTXT67,'') as [ANSWER_2], -- captured by the agent if SPARE_FIELD_2 populated
isnull(c.attTXT37,'') as [QUESTION_3], -- populate from SPARE_FIELD_28
isnull(c.attTXT58,'') as [ANSWER_3] -- captured by the agent if SPARE_FIELD_3 populated
--isnull(c.attdt18,GETDATE()) as [Export_Date]
from
u_contact c with (nolock)
inner join u_campaigncontact uc with (nolock) on uc.contactid = c.id
inner join u_dispcode d with (nolock) on d.id = uc.resultcodeid
outer apply (select top 1 duration, callid,
case when outcome = 113 then 1 else 0 end [Abandon], [datetime] from dxi_cdrlog cdr where cdr.urn = c.id order by callid desc) dx
where
uc.campaignid in (3, 18, 22, 26) and d.dmc = 1 and c.created between DATEADD(week, -1, getdate()) AND getdate()
and c.importid > 0
Since you are using SQL-Server you could use the command line tool sqlcmd which comes with the package. You might have to re-run the installation of the SSMS to acticvate the option.
Open cmd.exe. When you use the sqlcmd command with the options
sqlcmd -S host -d dbname -U username -P password -W -s ~ -h -1 -Q "SET NOCOUNT ON;SELECT 1,'hello world'"
It should get you
1~hello world
-W removes blanks between the columns, -s ~ sets ~ as the column separator, -h -1 removes the header line at the top and -Q then expects the actual query statement as the next argument.
Obviously for the actual job you need to call your SELECt statement. It might be a good idea to define a view (myview) for the job and in the sqlcmd just do a SELECT * FROM myview since writing a multiline SQL command within a cmd environment is not so much fun .... Of course it is possible but it is not really worth the hassle.
The command should then redirect its output directly into a file like
set sql=SET NOCOUNT ON;SELECT * FROM myview
set scmd=sqlcmd -S host -d dbname -U username -P password -W -s ~ -h -1 -Q
%scmd% "%sql" > exportfile.csv
I have downloaded the GeoLiteCountry CSV file from Maxmind - http://www.maxmind.com/app/geolitecountry. Using the format given to me as standard (so that this can become an automated task) I am attempting import all the data into a table.
I created a new table IPCountries2 which has columns exactly matching the columns provided:
FromIP varchar(50),
ToIP varchar(50),
BeginNum bigint,
EndNum bigint,
CountryCode varchar(50),
CountryName varchar(250)
Using the various chunks of code I could find, I was unable to get it working using the field terminator and row terminator:
BULK
INSERT CSVTest
FROM 'c:\csvtest.txt'
WITH
(
FIELDTERMINATOR = '","',
ROWTERMINATOR = '\n'
)
GO
The result of this was a single row inserted, all correct except the last one had overflowed with the next lines (presumably the whole database if I didn't have a limit). Also, the first cell had a quote at the start.
I looked around and found something called a format file (never used these). Made one which looks like:
10.0
6
1 SQLCHAR 0 50 "," 1 FromIP ""
2 SQLCHAR 0 50 "," 2 ToIP ""
3 SQLBIGINT 0 19 "," 3 BeginNum ""
4 SQLBIGINT 0 19 "," 4 EndNum ""
5 SQLCHAR 0 50 "," 5 CountryCode ""
6 SQLCHAR 0 250 "\n" 6 CountryName ""
but this errors on the bigint lines:
Msg 4867, Level 16, State 1, Line 1
Bulk load data conversion error (overflow) for row 1, column 3 (BeginNum).
It does that 10 times and then stops because of maximum error count.
I was able to get the first method working if I took it into Excel and re-saved, this removed the quotes. However, I don't want to rely on this method as I want this to update automatically every week and not have to open and re-save manually.
I don't mind which of the two methods I use ultimately, just so long as it works with a clean file. I had a look at their documentation but they only have code for PHP or MS Access.
Edit
Some lines from the CSV file:
"1.0.0.0","1.0.0.255","16777216","16777471","AU","Australia"
"1.0.1.0","1.0.3.255","16777472","16778239","CN","China"
"1.0.4.0","1.0.7.255","16778240","16779263","AU","Australia"
"1.0.8.0","1.0.15.255","16779264","16781311","CN","China"
"1.0.16.0","1.0.31.255","16781312","16785407","JP","Japan"
"1.0.32.0","1.0.63.255","16785408","16793599","CN","China"
"1.0.64.0","1.0.127.255","16793600","16809983","JP","Japan"
"1.0.128.0","1.0.255.255","16809984","16842751","TH","Thailand"
"1.1.0.0","1.1.0.255","16842752","16843007","CN","China"
"1.1.1.0","1.1.1.255","16843008","16843263","AU","Australia"
"1.1.2.0","1.1.63.255","16843264","16859135","CN","China"
"1.1.64.0","1.1.127.255","16859136","16875519","JP","Japan"
"1.1.128.0","1.1.255.255","16875520","16908287","TH","Thailand"
Update
After some persisting I was able to get things working 95% with the original method (without the format document). However, it was changed slightly to look like so:
BULK INSERT IPCountries2
FROM 'c:\Temp\GeoIPCountryWhois.csv'
WITH
(
FIELDTERMINATOR = '","',
ROWTERMINATOR = '"'
)
GO
Everything goes in the right fields as they should, the only issue I have is in the first column there is a quote at the beginning. Some sample data:
FromIP ToIP BeginNum EndNum CountryCode Country
"2.21.248.0 2.21.253.255 34994176 34995711 FR France
"2.21.254.0 2.21.254.255 34995712 34995967 EU Europe
"2.21.255.0 2.21.255.255 34995968 34996223 NL Netherlands
Success. Searching around and some help from another forum finally got me to my solution. For those in need of a similar solution, keep reading:
I ended up using the format file method - whether it would be possible to use fieldterminators and row terminators I'm not sure.
My SQL code looks like:
CREATE TABLE #TempTable
(
DuffColumn varchar(50),
FromIP varchar(50),
ToIP varchar(50),
BeginNum bigint,
EndNum bigint,
CountryCode varchar(50),
CountryName varchar(250)
)
BULK
INSERT #TempTable
FROM 'c:\Temp\GeoIPCountryWhois.csv'
WITH
(
FORMATFILE = 'C:\Temp\format.fmt'
)
INSERT INTO IPCountries2 (FromIP, ToIP, BeginNum, EndNum, CountryCode, Country)
SELECT FromIP, ToIP, BeginNum, EndNum, CountryCode, CountryName FROM #TempTable
As found in my research, it was necessary to have a useless column which simply captured the first quote.
My format file looks like:
10.0
7
1 SQLCHAR 0 1 "" 1 DuffColumn ""
2 SQLCHAR 0 50 "\",\"" 2 FromIP ""
3 SQLCHAR 0 50 "\",\"" 3 ToIP ""
4 SQLCHAR 0 19 "\",\"" 4 BeginNum ""
5 SQLCHAR 0 19 "\",\"" 5 EndNum ""
6 SQLCHAR 0 50 "\",\"" 6 CountryCode ""
7 SQLCHAR 0 250 "\"\n" 7 CountryName ""
To note, despite eventually being stored as a BIGINT, BeginNum and EndNum are both passed in as SQLCHARS, otherwise the insert does an odd multiplication on the numbers (something about reading it as bytes rather than digits, I didn't entirely understand it).
And that's about it. The last thing to automate this script fully is to truncate the table first so as to clear out old records. However that might not be to everyones needs.
Try this command. All I did is remove the double quotes from your FIELDTERMINATOR:
BULK
INSERT CSVTest
FROM 'c:\csvtest.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
Your data fields are actually terminated by commas, not commas wrapped in quotes. I also suggest building a staging/import table match the datatypes of your source file exactly, which in this case would look like:
FromIP varchar(50),
ToIP varchar(50),
BeginNum varchar(50),
EndNum varchar(50),
CountryCode varchar(50),
CountryName varchar(250)
Your source data for BeginNum and EndNum is actually string, not bigint. You can convert this data once you have it imported into your staging table.
declare #sql varchar(1000)
declare #filename varchar(100) = 'C:\Temp\GeoIPCountryWhois.csv'
set #sql =
'BULK INSERT geoip FROM ''' + #filename + '''
WITH
(
CHECK_CONSTRAINTS,
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''' + char(0x0A) + '''
)'
exec (#sql)