How does one write, using SQLCMD:
multiple result sets to one output file?
or, multiple result sets to separate output files?
Discussion
After prototyping in SSMS, then moving to SQLCMD called from batch file, it's necessary to stay within the same connection (due to building some #temp tables along the way). The batch files will then be provided to production operations who will run them and give the output back to me for further processing.
CREATE TABLE #BatchFileType ( ... )
INSERT INTO #BatchFileType ( ... )
SELECT (...) FROM ...
CREATE NONCLUSTERED INDEX ...
CREATE TABLE #BrandingServiceDates ( ... )
INSERT INTO #BrandingServiceDates (
SELECT (...) FROM #BatchFileType JOIN (other tables)
CREATE NONCLUSTERED INDEX ...
SELECT [result set 1]
SELECT [result set 2]
SELECT [result set n]
...
Then, based on #BrandingServiceDates, create multiple result sets which we want to write to output files. Each run is based on a date range. The goal here is to not have to redo the #temp table processing time for each result set.
This is a one-time run so looking to solve this with sqlcmd, batch files, and parameters.
sqlcmd -S %1 -i %2 -W -o %3 -k -s,
Where -o (from what I can tell) doesn't take an array of filenames.
Alternatives to SQLCMD are also welcome.
I have done this before by using xp_cmdshell and sqlcmd to create delimited files. This example used a trusted connection back to the server that made the call. If there are any errors check the results written to #output. Or possibly the errors are written to the TXT file.
-- xp_cmdshell has a 8000 charater limit
declare #cmd varchar(8000)
-- create a global ##temp table
-- command string to output the table to a couple of different files
set #cmd = 'sqlcmd -E -S "[server]" -d "[database]" -Q"SET NOCOUNT ON select * from ##temp" -s"[delimiter]" -W -h-1 > "d:\file1.txt"'
+ ' & sqlcmd -E -S "[server]" -d "[database]" -Q"SET NOCOUNT ON select * from ##temp" -s"[delimiter]" -W -h-1 > "d:\file2.txt"'
-- replace dynamic values in the command string
set #cmd = replace(#cmd, '[server]', #server)
set #cmd = replace(#cmd, '[database]', #database)
set #cmd = replace(#cmd, '[delimiter]', #delimit)
-- execute the command and output results to a table variable
declare #output table (output varchar(max) null)
insert #output exec master..xp_cmdshell #cmd
Related
I'm trying to feed in a variable name from the command line and create a database if it does not exist. My command line is below:
sqlcmd -S localhost -i 00_SpinUp.sql -v DBName = TEST -o Script00.txt -b
where 00_SpinUp.sql is as follows:
DECLARE #DBNAME VARCHAR(MAX);
SET NOCOUNT ON
GO
IF DB_ID('$(DBNAME)') IS NULL
BEGIN
CREATE DATABASE #DBNAME
END
Yet I'm getting a syntax error. What have I done wrong?
With SQLCMD variables, reference the variable in the script with $(VariableName) rather than as T-SQL variables. No declaration is needed in the script.
SET NOCOUNT ON;
GO
IF DB_ID('$(DBNAME)') IS NULL
BEGIN
CREATE DATABASE [$(DBNAME)];
END;
I am trying to execute some sqlcmd through T-SQL on SQL Server 2008. There is a part of my code where I am checking a data file size and if that data file size does not equal to 0, then start deleting the specific table so I can BCP in new data.
Below is my code that is not being executed:
SET #myVariable = '
SETLOCAL
FOR %%R IN (X:\Main Folder\Data\'+#databaseName+'_'+#tableName+'.dat) DO SET size=%%~zR
IF %size% NEQ 0 (
SQLCMD -E -S my-server-name -Q "DELETE FROM '+#databaseName+'.'+#schemaName+'.'+#tableName+';" >> X:\Main Folder\Log\Log.txt
)'
EXEC master..xp_cmdshell #myVariable
For some reason when I execute my stored procedure, the code above seems to be skipped because it does not shoot back any error messages.
EDIT: After re-adjusting the spacing and my code, #myVariable, gets executed now. However, it still does not work in regards that it still deletes the table even though the data file size = 0. However, when I hard code it within a batch file, it works perfectly fine. Any ideas?
You need to use single % in your for loop as you are not executing the code in a batch file (that requires %%), see this post for some further clarification. So your for loop should be:
FOR %R IN (X:\Main Folder\Data\'+#databaseName+'_'+#tableName+'.dat) DO SET size=%~zR
I think the problem is that you're not using quotes around your filenames. That top level directory has a space in it.
Jaco's answer looks correct and I'm sure it was part of the problem. You should probably initialize size just to be safe too:
SET #myVariable = '
SETLOCAL
SET size=0
FOR %R IN ("X:\Main Folder\Data\'+#databaseName+'_'+#tableName+'.dat") DO SET size=%~zR
IF %size% NEQ 0 (
SQLCMD -E -S my-server-name -Q "DELETE FROM '+#databaseName+'.'+#schemaName+'.'+#tableName+';" >> "X:\Main Folder\Log\Log.txt"
)'
EXEC master..xp_cmdshell #myVariable
Without the quotes the for loop is going to treat that its "set" (the terminology used in for /?) as two items separated by the space. If your current directory is X:\Main Folder\Data\ it would still work though since it sees the last one as a relative path to the .dat file and then still sets the right value on the last pass.
Why would you go 'down' to the command line at all? (there are reasons why xp_cmdshell is disabled by default)
Could you not simply loop over your tables (sys.tables WHERE name LIKE ...) and then
create a shadow-copy of the table (SELECT INTO)
BULK INSERT from the (expected) file into shadow-table (in a TRY..CATCH to handle situations where the file does not exist, or is empty, or is corrupt, ..
if there is data in the shadow-table, then DELETE the actual table records and move the data over
if there is no data in the shadow table then DELETE the actual table records (or leave them in if you assume a missing or empty bcp file means it will arrive later on and you're stuck with the current version for now)
DROP the shadow table again
Not sure if this is an option for you but since you are on SQL 2008 you should be able to use powershell command instead:
DECLARE #File varchar(100), #outputFile varchar(100)
DECLARE #cmd varchar(1000)
SELECT #File = 'path_to_file'
SELECT #outputFile = 'path_to_output_file'
SELECT #cmd = 'powershell.exe -command "if((Get-Item '''+#File+''').length -gt 0) {&sqlcmd.exe -E -S SERVERNAME -Q ''SELECT name FROM master.sys.databases ;'' -o '+#outputFile+'}"'
SELECT #cmd
exec master..xp_cmdshell #cmd
I've checked and it seems to be working depending on the file size.
I can't speak to your DOS command. I can however suggest using Ole Automation Procedures to get the file size. That way you would not have to rely on running batch commands.
First you need to enable Ole Automation Procedures on your SQL Server instance, as follows:
sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'Ole Automation Procedures', 1;
GO
RECONFIGURE;
GO
You only need to do this once.
Next is a script that gets the file size. The example assumes that there's a file called C:\Temp\testfile.txt. The script selects the size if the file exists, or selects 0 if it doesn't. You can take this script as an example to do what you want based on the size.
Here goes:
DECLARE #hr INT;
DECLARE #size INT;
DECLARE #obj_file INT;
DECLARE #obj_file_system INT;
DECLARE #file_name VARCHAR(100)='C:\Temp\testfile.txt';
-- Create a FileSystemObject. Create this once for all subsequent file manipulation. Don't forget to destroy this object once you're done with file manipulation (cf cleanup)
EXEC #hr = sp_OACreate 'Scripting.FileSystemObject', #obj_file_system OUT;
IF #hr<>0 GOTO __cleanup;
-- Get a handle for the file. Don't forget to release the handle for each file you get a handle for (see cleanup). The return will be different from 0 if the file doesn't exist
EXEC #hr = sp_OAMethod #obj_file_system, 'GetFile', #obj_file out, #file_name;
IF #hr<>0 GOTO __print_file_size;
-- Retrieve the file size.
EXEC sp_OAGetProperty #obj_file, 'size', #size OUT;
__print_file_size:
SELECT ISNULL(#size,0) AS file_size;
__cleanup:
EXEC sp_OADestroy #obj_file_system;
EXEC sp_OADestroy #obj_file;
You are using X:\ in your code. But the code is running under the service account for SQL Server. That account may not have x: available.
I would suggest using a UNC instead of a mapped drive. Also, make sure that your service is running under a domain account, and that the domain account has all required permissions to the UNC.
I realized that I can check the table count instead of a data file size by using this method:
SET #sqlCheck = 'SQLCMD -E -S ServerA -Q "IF (SELECT COUNT(*) FROM '+#databaseName+'.'+#schemaName+'.'+#tableName+') > 0 BEGIN DELETE FROM ServerB.'+#databaseName+'.'+#schemaName+'.'+#tableName+' END;"'
EXEC MASTER..xp_cmdshell #sqlcheck
You seems to know the names of the databases and tables already, so you can use the following, which basically does a DIR for the file you're looking for and checks if it's '0 bytes', if so it then does whatever you want. Things to note:
STRING TEMPLATES -- When building strings, I like to build a 'template' and then replace within the string. This is a good way to make sure you have the right number of quotes, parenthesis, etc. I did it twice here, once to build the DIR command and then again to build the TRUNCATE command.
TRUNCATE -- although not part if your question, you may want to use a TRUNCATE instead of DELETE FROM. If you had a million rows in your table, DELETE FROM may take 2 min to run, where as TRUNCATE will always take 0-seconds to run.
Your answer:
SET NOCOUNT ON;
DECLARE #DatabaseName VARCHAR(50) = 'database1'
DECLARE #TableName VARCHAR(50) = 'table1'
DECLARE #PathTemplate VARCHAR(50) = 'dir c:\temp\{#DatabaseName}_{#TableName}.txt'
SET #PathTemplate = REPLACE(#PathTemplate, '{#DatabaseName}', #DatabaseName);
SET #PathTemplate = REPLACE(#PathTemplate, '{#TableName}', #TableName);
DECLARE #FileNames AS TABLE (FileNames VARCHAR(100))
INSERT #FileNames (FileNames)
exec xp_cmdshell #PathTemplate
IF EXISTS ( SELECT 1 FROM #FileNames WHERE FileNames LIKE '%0 bytes')
BEGIN
PRINT 'No Content/Missing File'
END
ELSE BEGIN
DECLARE #SqlExc VARCHAR(500) = 'TRUNCATE TABLE [{#DatabaseName}].[dbo].[{#TableName}]'
SET #SqlExc = REPLACE(#SqlExc, '{#DatabaseName}', #DatabaseName);
SET #SqlExc = REPLACE(#SqlExc, '{#TableName}', #TableName);
PRINT #SqlExc
-- sp_executesql #SqlExc <-- Do this in production
END
I have a SQL script I need to run on about 20 different databases.
I basically just need to be able to run some SQL, then have it load and run a file from the disk, do more SQL, run that same script again, etc.
I was hoping to make a SQL script that would basically look something like this:
use database1
go
exec c:\release.sql
go
use database2
go
exec c:\release.sql
go
use database3
go
exec c:\release.sql
go
--etc....
I've looked online a bunch and found a way to do something similar in a batch file with sqlcmd but it isn't working and I don't see how to switch databases that way, either.
Thanks a ton!
Ben
You can switch management studio to sqlcmd mode (query menu) and then run a script with :r script.sql
To do this on a dynamically generated list of databases you have to do some sqlcmd trickery:
set output to file
generate the command to execute
set output to stdout
execute the file
delete the temp file
I assume in this example that the file script.sql exists in c:\temp. Note that the GO statements are important in the script or the sqlcmd parser will get confused.
:OUT $(TEMP)\db.sql
declare #script nvarchar(max)
select #script = isnull(#script, '')
+ 'use ' + name + char(13) + char(10)
+ ':r c:\temp\script.sql' + char(13) + char(10)
from sys.databases
where name like N'%[_]db'
print #script
GO
:OUT stdout
:r $(TEMP)\db.sql
GO
!!del $(TEMP)\db.sql /s /q
You don't need to do this in SSMS. You just need to create a CMD script.
IF you have a static set of databases to run on, then use the following:
#ECHO OFF
SET MyServer="(local)"
SET MyScript="c:\release.sql"
SQLCMD -S %MyServer% -E -i %MyScript% -d database1
SQLCMD -S %MyServer% -E -i %MyScript% -d database2
...
SQLCMD -S %MyServer% -E -i %MyScript% -d database20
IF you have a dynamic set of databases that can be queried for, then use the following:
#ECHO OFF
SET MyServer="(local)"
SET MyScript="c:\release.sql"
SET MyQuery="SET NOCOUNT ON; SELECT [Name] FROM [sys].[databases] sd WHERE sd.[name] LIKE N'%%[_]db' ORDER BY sd.[name];"
FOR /F %%B IN ('SQLCMD -h -1 -S %MyServer% -E -Q %MyQuery%') DO (
REM remove the "echo" from the next line to run the scripts
echo SQLCMD -S %MyServer% -E -i %MyScript% -d %%B -o results-%%B.txt
)
Using the %%B in the output filename will give you a different output file per database, as in:
results-database1_db.txt
results-database2_db.txt
...
Other notes:
Use (local) instead of localhost when connecting to the local, default instance as it uses shared memory while localhost forces a TCP connection.
If you are searching for an underscore in a LIKE statement, enclose it in square brackets else it is a single-character wild card (which still technically works sometimes, but could also match other characters): [_]
Thanks everyone who pitched in! The following seems like it might work (based on #srutzky's answer)
sqlcmd -S "localhost" -E -i "c:\release.sql" -d database1 -o results.txt
The thing I am missing by using a cmd prompt instead of SSMS is that I don't think I can write cursor to loop through each database that ends with "_db" and then execute against that... Here's the SQL I have but I just need to be able to put the link to the SQL file to execute.
link
If I put the release script SQL into this file into the #text variable it doesn't work because it blows up on each GO statement I have in my release.sql file.
declare #text as nvarchar(max)
set #text = N'
-- GET AND RUN SCRIPT FROM DISK!
'
declare C_CURSOR CURSOR FOR
select [Name] from sys.databases
where name like '%_db'
order by name
declare #runtext as nvarchar(max)
declare #DB_Name as nvarchar(200)
OPEN C_CURSOR
fetch next from C_CURSOR INTO #DB_Name
WHILE(##FETCH_STATUS = 0)
BEGIN
print #DB_Name
set #runtext = 'select ''' + #DB_Name + ''' as DatabaseName
use ' + #DB_Name + N'
' + #text
exec sp_executesql #runtext
fetch next from C_CURSOR INTO #DB_Name
END
CLOSE C_CURSOR
DEALLOCATE C_CURSOR
Thanks again!
I ended up combining 2 things. I made a SQL script that creates a cursor to find the databases and then prints a list of commands for a CMD prompt. I then run that in the command prompt. Below is what we output with our sql script and then save as a .bat file that we run. It's working great!
That script is essentially created with the following SQL script:
/*** GET DATABASES IN THE CURSOR QUERY BELOW! */
declare C_CURSOR CURSOR FOR
select [Name] from sys.databases
where name like '%_db'
order by name
/* THIS IS WHERE THE CURSOR STARTS*/
declare #DB_Name as nvarchar(200)
OPEN C_CURSOR
fetch next from C_CURSOR INTO #DB_Name
WHILE(##FETCH_STATUS = 0)
BEGIN
print 'SQLCMD -S "localhost" -E -i "C:\release.sql" -d ' + #DB_Name + ' -o ' + #DB_Name + '_results.txt'
fetch next from C_CURSOR INTO #DB_Name
END
CLOSE C_CURSOR
DEALLOCATE C_CURSOR
That outputs the following which we then run in a .bat file
SQLCMD -S "localhost" -E -i "C:\release.sql" -d database1 -o database1_results.txt
SQLCMD -S "localhost" -E -i "C:\release.sql" -d database2 -o database2_results.txt
SQLCMD -S "localhost" -E -i "C:\release.sql" -d database3 -o database3_results.txt
Thanks everyone!
I've spent all day scouring the net on answers. Apparently tsql doesn't have its own nifty write to file commands. Here is my dilemma
I have a load file that I am creating where a single line can reach 10K+ in length. On SQL Server varchar(MAX) limit is 8000 (so I believe) so I broke those lines into several variables. I tried to do PRINT but the window pane has allows 4000. The workaround is to print those broken lines one variable at a time but that can get tedious for manual labor so I opted to look into writing it into a txt file one variable at a time.
I looked into BCP via xpcommandshell and it looked promising. Issue was that I could get this line to work on the command prompt yet that exact same line doesn't work on TSQL query:
declare #cmd varchar(8000)
select #cmd = 'bcp Client_DB "Select name from dbo.t_TagBuild_set" queryout "Desktop\LAMB\dummy.txt" -c -t, -T'
exec master..xp_cmdshell #cmd
bcp Client_DB "Select name from dbo.t_TagBuild_set" queryout "Desktop\LAMB\dummy.txt" -c -t, -T works perfectly fine on command prompt
despite this slight progress, my manager didn't want to go that route. So instead I opted for sp_OACreate and sp_OAMethod after enabling sp_configure via executing this line on SQL:
sp_configure 'Ole Automation Procedures', 1
One of the very first lines on this route is this:
EXECUTE #hr = sp_OACreate 'Scripting.FileSystemObject' , #objFileSystem OUT
#hr gives a 0 so that's good but #objFileSystem yields 16711422 and #hr eventually becomes -2146828218 which i believe is permissions.
i really am at a loss on finding something simple to do yet i've made this increasingly difficult on myself to find something concrete just to write to a couple variables in a row before adding a new line and repeat the process.
If anyone can expertly help me figure out BCP or sp_OACreate then I'd be very appreciative cause the net as is barely helps (and this is after I spent a lot of time looking through Microsofts own site for an answer)
The reason your BCP didn't work is because you were running it from xp_cmdshell with a trusted user. xp_cmdshell is not run under the user running the script. You can either a) change your bcp command to use a sql login/password or b) create a job to run it (not xp_cmdshell) because you can control what user it is run as by using run as and a credential. You can then launch the job within a script by using sp_start_job.
Your other good option is to create an SSIS package and either run it through the command line (say in a bat file) or again run it through a job.
Create a view of your query and select it using sqlcmd.
declare #cmd varchar(8000)
select #cmd = 'sqlcmd -h-1 -W -S servername -d database -U username -P password -Q "SET NOCOUNT ON; select * from VIEW_NAME " -o "C:\OUTPUT\query.csv" '
exec master..xp_cmdshell #cmd
-h-1 removes the header
SET NOCOUNT ON removes the rows affected footer
You can write to file on T-SQL using this (it works into trigger):
--char(9) = \t
DECLARE #filename nvarchar(1000);
SET #filename = ' (echo '+#parameterA+ char(9) +#parameterB+ ... + char(9) +#ParameterN+') > e:\file1.txt && type e:\file1.txt >> e:\file2.txt';
exec DatabaseName..xp_cmdshell #filename, no_output
I am trying to pass in double quote to a scripting variable in SQLCMD. Is there a way to do this?
sqlcmd -S %serverName% -E -d MSDB -i MyScript.sql -m 1 -v Parameter="\""MyValueInDoubleQuote\"""
And my sql script is as follow:
--This Parameter variable below is commented out since we will get it from the batch file through sqlcmd
--:SETVAR Parameter "\""MyValueInDoubleQuote\"""
INSERT INTO [MyTable]
([AccountTypeID]
,[Description])
VALUES
(1
,$(Parameter))
GO
If you have your sql script set up in this fashion:
DECLARE #myValue VARCHAR(30)
SET #myValue = $(MyParameter)
SELECT #myValue
Then you can get a value surrounded by double quotes into #myValue by just enclosing your parameter in single quotes:
sqlcmd -S MyDb -i myscript.sql -v MyParameter='"123"'
This works because -v is going to replace the $(MyParameter) string with the text '"123"'. The resulting script will look like this before it is executed:
DECLARE #myValue VARCHAR(30)
SET #myValue = '"123"'
SELECT #myValue
Hope that helps.
EDIT
This sample is working for me (tested on SQL Server 2008, Windows Server 2K3). It inserts a record into the table variable #MyTable, and the value in the Description field is enclosed in double quotes:
MyScript.sql (no need for setvar):
DECLARE #MyTable AS TABLE([AccountTypeID] INT, [Description] VARCHAR(50))
INSERT INTO #MyTable ([AccountTypeID] ,[Description])
VALUES(1, $(Parameter))
SELECT * FROM #MyTable
SQLCMD:
sqlcmd -S %serverName% -E -d MSDB -i MyScript.sql -m 1 -v Parameter='"MyValue"'
If you run that script, you should get the following output, which I think is what you're looking for:
(1 rows affected)
AccountTypeID Description
------------- --------------------------------------------------
1 "MyValue"
Based on your example, you don't need to include the quotes in the variable, as they can be in the sql command, like so:
sqlcmd -S %serverName% -E -d MSDB -i MyScript.sql -m 1 -v Parameter="MyValueNoQuotes"
and
INSERT INTO [MyTable]
([AccountTypeID]
,[Description])
VALUES
(1
,"$(Parameter)")
(Though I am more accustomed to use single quotes, as in ,'$(Parameter)'