I am having an issue with setting a double quote (") text delimter to my .csv file using BCP.
Take for example the current SQL script;
declare #sql varchar(8000)select #sql = 'bcp dmP.pdiStb.vTest2 out
c:\bcp\test001234.csv -c -t"\",\"" -T -S'+ ##servername
EXEC master..xp_cmdshell #sql
When I execute the above, I get a CSV file that looks like the following;
55","sql_gmi_srv","Policy_RenewalDueDate
The above is missing a double quote at the start and end of the file, is there a solution to solve this?
Related
I am dumping tables from a SQL Server 2016 instance with bcp - using this method to generate newline delimited json exports. These files are encoded as UTF-16LE, but to import them into BigQuery I need to convert them to UTF-8.
Using iconv -f utf-16le -t UTF-8 table.json > table.utf8.json works in 99% of the cases - but I am stumbling over emojis in the files (from user input in a web form). Some of them cannot be converted and iconv errors out with:
iconv: table.json:39972:807: cannot convert
Checking the file yields some emojis and some undefined chars:
Deleting those chars, lets iconv finish the job:
How can I either export valid UTF-8 from SQL-Server or convert the resulting files into UTF-8 if I want to keep the emojis? Is it possible that SQL Server stores incorrectly encoded chars?
My current workaround is to use the -c switch of iconv, to discard these characters silently.
Update:
This is the current script to extract the JSON files from SQL Server:
set #cmd ='bcp "select '
IF #top is not null
set #cmd = #cmd +' top('+#top+') '
IF #startdate is null
BEGIN
set #filename = (SELECT replace(#currenttable,'.','_'))
set #cmd = #cmd + '(select '+#header+' for json path, '
set #cmd = #cmd+ ' without_array_wrapper) from DWH.'+#currenttable+' " queryout Z:\json\'+#filename+'.json -a65535 -w -T -S SRV\DWH -d DWH'
END
ELSE
BEGIN
set #filename = (SELECT replace(replace(replace(#currenttable,']',''),'[',''),'.','_')+'_'+#currentdateid)
set #cmd = #cmd + '(select '+#header+' for json path, '
set #cmd = #cmd+ 'without_array_wrapper) from DWH.'+#currenttable+' where '+#partitioncolumn+'>= '''+#startdate+''''
set #cmd = #cmd+ ' AND '+#partitioncolumn+'< '''+#enddate+'''"'
set #cmd = #cmd+ ' queryout Z:\json\'+#filename+'.json -a65535 -w -T -S SRV -d DWH'
END
exec xp_cmdshell #cmd, no_output
I have a SQL script I need to run on about 20 different databases.
I basically just need to be able to run some SQL, then have it load and run a file from the disk, do more SQL, run that same script again, etc.
I was hoping to make a SQL script that would basically look something like this:
use database1
go
exec c:\release.sql
go
use database2
go
exec c:\release.sql
go
use database3
go
exec c:\release.sql
go
--etc....
I've looked online a bunch and found a way to do something similar in a batch file with sqlcmd but it isn't working and I don't see how to switch databases that way, either.
Thanks a ton!
Ben
You can switch management studio to sqlcmd mode (query menu) and then run a script with :r script.sql
To do this on a dynamically generated list of databases you have to do some sqlcmd trickery:
set output to file
generate the command to execute
set output to stdout
execute the file
delete the temp file
I assume in this example that the file script.sql exists in c:\temp. Note that the GO statements are important in the script or the sqlcmd parser will get confused.
:OUT $(TEMP)\db.sql
declare #script nvarchar(max)
select #script = isnull(#script, '')
+ 'use ' + name + char(13) + char(10)
+ ':r c:\temp\script.sql' + char(13) + char(10)
from sys.databases
where name like N'%[_]db'
print #script
GO
:OUT stdout
:r $(TEMP)\db.sql
GO
!!del $(TEMP)\db.sql /s /q
You don't need to do this in SSMS. You just need to create a CMD script.
IF you have a static set of databases to run on, then use the following:
#ECHO OFF
SET MyServer="(local)"
SET MyScript="c:\release.sql"
SQLCMD -S %MyServer% -E -i %MyScript% -d database1
SQLCMD -S %MyServer% -E -i %MyScript% -d database2
...
SQLCMD -S %MyServer% -E -i %MyScript% -d database20
IF you have a dynamic set of databases that can be queried for, then use the following:
#ECHO OFF
SET MyServer="(local)"
SET MyScript="c:\release.sql"
SET MyQuery="SET NOCOUNT ON; SELECT [Name] FROM [sys].[databases] sd WHERE sd.[name] LIKE N'%%[_]db' ORDER BY sd.[name];"
FOR /F %%B IN ('SQLCMD -h -1 -S %MyServer% -E -Q %MyQuery%') DO (
REM remove the "echo" from the next line to run the scripts
echo SQLCMD -S %MyServer% -E -i %MyScript% -d %%B -o results-%%B.txt
)
Using the %%B in the output filename will give you a different output file per database, as in:
results-database1_db.txt
results-database2_db.txt
...
Other notes:
Use (local) instead of localhost when connecting to the local, default instance as it uses shared memory while localhost forces a TCP connection.
If you are searching for an underscore in a LIKE statement, enclose it in square brackets else it is a single-character wild card (which still technically works sometimes, but could also match other characters): [_]
Thanks everyone who pitched in! The following seems like it might work (based on #srutzky's answer)
sqlcmd -S "localhost" -E -i "c:\release.sql" -d database1 -o results.txt
The thing I am missing by using a cmd prompt instead of SSMS is that I don't think I can write cursor to loop through each database that ends with "_db" and then execute against that... Here's the SQL I have but I just need to be able to put the link to the SQL file to execute.
link
If I put the release script SQL into this file into the #text variable it doesn't work because it blows up on each GO statement I have in my release.sql file.
declare #text as nvarchar(max)
set #text = N'
-- GET AND RUN SCRIPT FROM DISK!
'
declare C_CURSOR CURSOR FOR
select [Name] from sys.databases
where name like '%_db'
order by name
declare #runtext as nvarchar(max)
declare #DB_Name as nvarchar(200)
OPEN C_CURSOR
fetch next from C_CURSOR INTO #DB_Name
WHILE(##FETCH_STATUS = 0)
BEGIN
print #DB_Name
set #runtext = 'select ''' + #DB_Name + ''' as DatabaseName
use ' + #DB_Name + N'
' + #text
exec sp_executesql #runtext
fetch next from C_CURSOR INTO #DB_Name
END
CLOSE C_CURSOR
DEALLOCATE C_CURSOR
Thanks again!
I ended up combining 2 things. I made a SQL script that creates a cursor to find the databases and then prints a list of commands for a CMD prompt. I then run that in the command prompt. Below is what we output with our sql script and then save as a .bat file that we run. It's working great!
That script is essentially created with the following SQL script:
/*** GET DATABASES IN THE CURSOR QUERY BELOW! */
declare C_CURSOR CURSOR FOR
select [Name] from sys.databases
where name like '%_db'
order by name
/* THIS IS WHERE THE CURSOR STARTS*/
declare #DB_Name as nvarchar(200)
OPEN C_CURSOR
fetch next from C_CURSOR INTO #DB_Name
WHILE(##FETCH_STATUS = 0)
BEGIN
print 'SQLCMD -S "localhost" -E -i "C:\release.sql" -d ' + #DB_Name + ' -o ' + #DB_Name + '_results.txt'
fetch next from C_CURSOR INTO #DB_Name
END
CLOSE C_CURSOR
DEALLOCATE C_CURSOR
That outputs the following which we then run in a .bat file
SQLCMD -S "localhost" -E -i "C:\release.sql" -d database1 -o database1_results.txt
SQLCMD -S "localhost" -E -i "C:\release.sql" -d database2 -o database2_results.txt
SQLCMD -S "localhost" -E -i "C:\release.sql" -d database3 -o database3_results.txt
Thanks everyone!
I am having issue with the way results are displayed with a bcp export to csv.
The below works fine, but is comma delimited so won't work for what I need.
DECLARE #sql VARCHAR(8000)
SET #sql = 'bcp "SELECT * FROM db..viewname" queryout "C:\test.csv" -c –t, -T -S <SERVERNAME>'
EXEC master..xp_cmdshell #sql
Results:
But If I change the "," to a pipe (or anything else) it breaks.
DECLARE #sql VARCHAR(8000)
SET #sql = 'bcp "SELECT * FROM db..viewname" queryout "C:\test.csv" -c –t"|" -T -S <SERVERNAME>'
EXEC master..xp_cmdshell #sql
Results:
The view used is a simple column select from a table with a WHERE clause.
What am I missing here..?
I know this is an old question but I just ran into this problem yesterday. To add a pipe delimiter you need to escape the pipe with a carrot like so:
-t^|
So set your #SQL like so:
SET #sql = 'bcp "SELECT * FROM db..viewname" queryout "C:\test.csv" -c –t^| -T -S <SERVERNAME>'
Hope this helps someone. :)
I've spent all day scouring the net on answers. Apparently tsql doesn't have its own nifty write to file commands. Here is my dilemma
I have a load file that I am creating where a single line can reach 10K+ in length. On SQL Server varchar(MAX) limit is 8000 (so I believe) so I broke those lines into several variables. I tried to do PRINT but the window pane has allows 4000. The workaround is to print those broken lines one variable at a time but that can get tedious for manual labor so I opted to look into writing it into a txt file one variable at a time.
I looked into BCP via xpcommandshell and it looked promising. Issue was that I could get this line to work on the command prompt yet that exact same line doesn't work on TSQL query:
declare #cmd varchar(8000)
select #cmd = 'bcp Client_DB "Select name from dbo.t_TagBuild_set" queryout "Desktop\LAMB\dummy.txt" -c -t, -T'
exec master..xp_cmdshell #cmd
bcp Client_DB "Select name from dbo.t_TagBuild_set" queryout "Desktop\LAMB\dummy.txt" -c -t, -T works perfectly fine on command prompt
despite this slight progress, my manager didn't want to go that route. So instead I opted for sp_OACreate and sp_OAMethod after enabling sp_configure via executing this line on SQL:
sp_configure 'Ole Automation Procedures', 1
One of the very first lines on this route is this:
EXECUTE #hr = sp_OACreate 'Scripting.FileSystemObject' , #objFileSystem OUT
#hr gives a 0 so that's good but #objFileSystem yields 16711422 and #hr eventually becomes -2146828218 which i believe is permissions.
i really am at a loss on finding something simple to do yet i've made this increasingly difficult on myself to find something concrete just to write to a couple variables in a row before adding a new line and repeat the process.
If anyone can expertly help me figure out BCP or sp_OACreate then I'd be very appreciative cause the net as is barely helps (and this is after I spent a lot of time looking through Microsofts own site for an answer)
The reason your BCP didn't work is because you were running it from xp_cmdshell with a trusted user. xp_cmdshell is not run under the user running the script. You can either a) change your bcp command to use a sql login/password or b) create a job to run it (not xp_cmdshell) because you can control what user it is run as by using run as and a credential. You can then launch the job within a script by using sp_start_job.
Your other good option is to create an SSIS package and either run it through the command line (say in a bat file) or again run it through a job.
Create a view of your query and select it using sqlcmd.
declare #cmd varchar(8000)
select #cmd = 'sqlcmd -h-1 -W -S servername -d database -U username -P password -Q "SET NOCOUNT ON; select * from VIEW_NAME " -o "C:\OUTPUT\query.csv" '
exec master..xp_cmdshell #cmd
-h-1 removes the header
SET NOCOUNT ON removes the rows affected footer
You can write to file on T-SQL using this (it works into trigger):
--char(9) = \t
DECLARE #filename nvarchar(1000);
SET #filename = ' (echo '+#parameterA+ char(9) +#parameterB+ ... + char(9) +#ParameterN+') > e:\file1.txt && type e:\file1.txt >> e:\file2.txt';
exec DatabaseName..xp_cmdshell #filename, no_output
Below when I executing the stored procedure in either code or in sql server management studio 2008 it saids it copied successfully rows, But I am not for sure why its not writing the file to disk. can some please help me, I have been struggling with this for the pass few days. I have enable command shell and everything. The weird thing is it works from the command line, but when i execute from stored procedure, it does not write file to disks, can some some help me
USE [ColorDb]
GO
ALTER PROCEDURE [dbo].[prc_WriteTableToTextDelimitedFile]
(
#FilePath VarChar(256)
)
AS
BEGIN
DECLARE #sql varchar(8000)
SET #sql = 'bcp "SELECT * FROM ColorDb.dbo.ColorTable" queryout "'+ #FilePath +'" -c -t; -T -SXXXXXXXXXXXX'
Print #sql
exec master..xp_cmdshell #sql
END
RETURN
I would print out the contents of #sql, and #FilePath.
I just ran some code through the MS SQL Server Manager Debugger window, you could step through the code. I did notice that you don't seem to have your quotes matched, and I don't see where you intialize #FilePath to anything. Looking at your last segment it looks like it has an unmatched double quote.
I didn't use any double quotes, I ran from a query in the query builder under MSSQL Server Manager. I got file output, moved files, concatenated files and what not from the query window.
Like this simple concatenation command:
Exec master..xp_cmdshell 'type c:\bcp\sysobjects.txt >> c:\bcp\a.txt'
Also this creation/output command:
SELECT #sql = 'bcp master..sysobjects out c:\bcp\sysobjects.txt -c -t\t -T -S' + ##servername
REPLACE( #sql, 'sysobjects.txt', #EmpRecordsFILENAME)
EXEC master..xp_cmdshell #sql
The files got created, and concatenated with these simple commands!