Exporting SQL Query results to Excel - sql-server

On executing the below mentioned statement:
EXEC proc_generate_excel_with_columns
'your dbname', 'your table name','your file path'
I'm getting the following error.Can anyone help?
User name not provided, either use -U to provide the user name or use
-T for Trusted Connection usage: bcp {dbtable | query} {in | out | queryout | format} datafile [-m maxerrors] [-f formatfile] [-e
errfile] [-F firstrow] [-L lastrow] [-b batchsize] [-n native type]
[-c character type] [-w wide character type] [-N keep non-text native]
[-V file format version] [-q quoted identifier] [-C code page
specifier] [-t field terminator] [-r row terminator] [-i inputfile]
[-o outfile] [-a packetsize] [-S server name] [-U username] [-P
password] [-T trusted connection] [-v version] [-R regional enable]
[-k keep null values] [-E keep identity values] [-h "load hints"] [-x
generate xml format file] [-d database name] NULL
My procedure is this:
create procedure proc_generate_excel_with_columns
(
#db_name varchar(100),
#table_name varchar(100),
#file_name varchar(100)
)
as
--Generate column names as a recordset
declare #columns varchar(8000), #sql varchar(8000), #data_file varchar(100)
select
#columns=coalesce(#columns+',','')+column_name+' as '+column_name
from
information_schema.columns
where
table_name='dbo.vcuriosoftronic.tblPayrollGroups'
select #columns=''''''+replace(replace(#columns,' as ',''''' as '),',',',''''')
--Create a dummy file to have actual data
select #data_file=substring('D:\TestFile.xls',1,len('D:\TestFile.xls')
-charindex('\',reverse('D:\TestFile.xls')))
+'D:\TestFile.xls'
--Generate column names in the passed EXCEL file
set #sql='exec master..xp_cmdshell ''bcp
" select * from (select '+#columns+') as t"
queryout "'+#file_name+'" -c'''
exec(#sql)
--Generate data in the dummy file
set #sql='exec master..xp_cmdshell ''bcp
"select * from [myserver]..'+#table_name+'"
queryout "'+#data_file+'" -c'''
exec(#sql)
--Copy dummy file to passed EXCEL file
set #sql= 'exec master..xp_cmdshell ''type '+#data_file+' >> "'+#file_name+'"'''
exec(#sql)
--Delete dummy file
set #sql= 'exec master..xp_cmdshell ''del '+#data_file+''''
exec(#sql)

Check out my blog article on how to use BCP to export data.
http://craftydba.com/?p=1690
The below snippet uses the -T, trusted connection. If you are running a job under the agent, it will run under that security account.
Please either pass the standard security credentials, -U -P or make sure the account has the ability to run the command.
-- BCP - Export query, pipe delimited format, trusted security, character format
DECLARE #bcp_cmd4 VARCHAR(1000);
DECLARE #exe_path4 VARCHAR(200) =
' cd C:\Program Files\Microsoft SQL Server\100\Tools\Binn\ & ';
SET #bcp_cmd4 = #exe_path4 +
' BCP.EXE "SELECT FirstName, LastName FROM AdventureWorks2008R2.Sales.vSalesPerson" queryout ' +
' "C:\TEST\PEOPLE.TXT" -T -c -q -t0x7c -r\n';
PRINT #bcp_cmd4;
EXEC master..xp_cmdshell #bcp_cmd4;
GO
Updated assuming the BCP path is in search list. Below is a screen shot with the path removed and the query changed for SQL Server 2012.
Look at the message window, it has the BCP command from the print statement. You can put the command into a batch file to test from the DOS prompt. It is a debugging exercise for you.

Related

Why do I get an error in bcp query out where clause?

I'm new to SQL Server and write this query for save select result into csv file:
declare #Cycle_ID as int
set #Cycle_ID = 0
EXECUTE master.dbo.xp_cmdshell 'bcp "select [Telno],[Cycle],[Price] FROM [ClubEatc].[dbo].[CycleAnalysisTable] where cast([Price] as float)>'+ #Cycle_ID +' " queryout d:\download\behi.csv -t"|" -c -S VM_TAZMINDARAMA -U behzad -P beh1368421'
In where clause I write simple variable, but I get this error:
Incorrect syntax near '+'.
Please don't decrease my question! I'm new! Thanks
SQL Server doesn't recognize expressions in exec statements. So, try setting up the query first in a variable and using that:
declare #Cycle_ID as int;
set #Cycle_ID = 0;
declare #sql nvarchar(max);
set #sql = '
bcp "select [Telno],[Cycle],[Price] FROM [ClubEatc].[dbo].[CycleAnalysisTable] where cast([Price] as float)>'+ cast(#Cycle_ID as varchar(255)) +' ";
EXECUTE master.dbo.xp_cmdshell #sql queryout d:\download\behi.csv -t"|" -c -S VM_TAZMINDARAMA -U behzad -P beh1368421';
It seems curious to me that you are comparing a column called Price to a variable called #Cycle_ID, but that has nothing to do with the syntax issue.

Using BCP to create XML files produces files with invalid encoding for non-ASCII characters

Reproducing issue, SQL code:
CREATE TABLE dbo.FooTable(SomeString NVARCHAR(100));
INSERT INTO dbo.FooTable(SomeString)
VALUES('Degree symbol is °');
DECLARE #Code NVARCHAR(4000) = N'BCP "SELECT (SELECT SomeString FROM dbo.FooTable FOR XML PATH(''Foo''), ROOT(''BAR''), TYPE )" QUERYOUT "F:\Output\File.XML" -c -C RAW -T ';
EXEC xp_cmdshell #Code;
DROP TABLE dbo.FooTable;
It creates file with following content:
<BAR><Foo><SomeString>Degree symbol is °</SomeString></Foo></BAR>
Such files are not recognized as valid XML files by Internet Explorer and Firefox (Chrome yields error error on line 1 at column 23: Encoding error). If I open them with Notepad and save as Unicode (little endian) - it opens and validates.
Update:
changing bcp options to -T -w -r -t seems to make XML valid for most XML consumers, but not Internet Explorer.
Try this one -
IF OBJECT_ID('tempdb.dbo.##t') IS NOT NULL
DROP TABLE ##t
SELECT x = (
SELECT x
FROM (
VALUES (N'Degree symbol is °')
) t(x)
FOR XML PATH('Foo'), ROOT('BAR'), TYPE
)
INTO ##t
DECLARE #sql NVARCHAR(4000) = 'bcp "SELECT * FROM ##t" queryout "D:\sample.xml" -S ' + ##servername + ' -T -w -r -t'
EXEC sys.xp_cmdshell #sql
Just interesting where the root of your issue...

SQL Server bcp utility does not create the txt file,

I'm new to this feature in SQL Server and could use some help. I'm experimenting with the BCP utility and the AdventureWorks2012 database.
I'm attempting to export data to a text file with the BCP utility and the code executes but a file is not created. Can you please look at my code and tell me where the problem(s) is/are.
I'm working out of a local copy of SQL Server Express. Thank you.
Declare #sql Varchar(8000)
Select #sql = 'bcp
+ SELECT FirstName, LastName
FROM AdventureWorks2012.Person.Person ORDER BY LastName, Firstname
+ queryout C:\Users\David\Desktop\yes.txt + -c -t, -T -S'
+ ##SERVERNAME
EXEC master..xp_cmdshell #sql
Here is my output when I run the query:
output
usage: bcp {dbtable | query} {in | out | queryout | format} datafile
[-m maxerrors] [-f formatfile] [-e errfile]
[-F firstrow] [-L lastrow] [-b batchsize]
[-n native type] [-c character type] [-w wide character type]
[-N keep non-text native] [-V file format version] [-q quoted identifier]
[-C code page specifier] [-t field terminator] [-r row terminator]
[-i inputfile] [-o outfile] [-a packetsize]
[-S server name] [-U username] [-P password]
[-T trusted connection] [-v version] [-R regional enable]
[-k keep null values] [-E keep identity values]
[-h "load hints"] [-x generate xml format file]
[-d database name] [-K application intent] [-l login timeout]
NULL
Here is the PRINT output:
bcp
+ "SELECT FirstName, LastName
FROM AdventureWorks2012.Person.Person ORDER BY LastName, Firstname"
+ queryout C:\Users\David\Desktop\yes.txt -c -t, -T -SHOMEPC\SQLINST01
TT's code worked. Here it is:
DECLARE #stmt_e VARCHAR(8000);
SET #stmt_e=
'BCP '+
'"SELECT FirstName,LastName FROM AdventureWorks2012.Person.Person ORDER BY LastName,Firstname" '+
'QUERYOUT "C:\Users\David\Desktop\yes.csv" '+
'-c -t, -T -S ' + ##SERVERNAME;
EXEC master.sys.xp_cmdshell #stmt_e;
The instructions for adding system permissions for database engine access can be found at the link below. I had to do this because my SQL Server Instance did not have permission to write to the path I was specifying.
https://msdn.microsoft.com/en-us/library/jj219062.aspx
The following snippet should run without problem on any SQL Server. It outputs all table information in INFORMATION_SCHEMA.TABLES as a comma separated file in C:\Temp\information_schema.csv.
Run this as a sanity check; it works without problem on my system, and it should on your system too. Run this from the AdventureWorks2012 database. If it doesn't work we'll have to delve deeper.
DECLARE #stmt_c VARCHAR(8000);
SET #stmt_c=
'BCP '+
'"SELECT*FROM '+QUOTENAME(DB_NAME())+'.INFORMATION_SCHEMA.TABLES" '+
'QUERYOUT "C:\Temp\information_schema.csv" '+
'-c -t, -T -S ' + ##SERVERNAME;
EXEC master.sys.xp_cmdshell #stmt_c;
Now if this works, adapt this to your query:
DECLARE #stmt_e VARCHAR(8000);
SET #stmt_e=
'BCP '+
'"SELECT FirstName,LastName FROM AdventureWorks2012.Person.Person ORDER BY LastName,Firstname" '+
'QUERYOUT "C:\Users\David\Desktop\yes.txt" '+
'-c -t, -T -S ' + ##SERVERNAME;
EXEC master.sys.xp_cmdshell #stmt_e;
This Should work
DECLARE #sql VARCHAR(8000);
SELECT #sql = 'bcp "SELECT FirstName, LastName FROM'+
' AdventureWorks2008.Person.Person ORDER BY FirstName, LastName" queryout'+
' C:\Users\David\Desktop\yes.txt -c -t, -r \r\n -S '+##servername+' -T';
EXEC master..xp_cmdshell #sql;
Should warpped the query in double quotes. I have removed an extra + before the -c.
You can test out the BCP on command prompt first. make sure it is working before using xp_cmdshell to execute it.
And lastly, i have added a PRINT statement to print out the command for verification
Declare #sql Varchar(8000)
Select #sql = 'bcp "SELECT FirstName, LastName '
+ 'FROM AdventureWorks2012.Person.Person '
+ 'ORDER BY LastName, Firstname" '
+ 'queryout C:\Users\David\Desktop\yes.txt -c -t, -T -S'
+ ##SERVERNAME
PRINT #sql -- Print out for verification
EXEC master..xp_cmdshell #sql
I have struggled with that problem almost whole yesterday and finally it comes out, that the "select" query is too complex (probably) to be processed by the xp_cmdshell command directly.
I have a query joining and aggregating many tables from different databases.
Trying to save its results to txt file directly via xp_cmdshell always resulted in the output presented by BrownEyeBoy, eventhought the select itself was working correctly.
I've bypassed this simply by inserting the results of the complex query into temporary table and then execute the xp_cmdshell on the temporary table like:
DECLARE
#SQL varchar(max)
, #path varchar(max) = 'D:\TMP\TMP_' + convert(varchar(10), convert(date, getdate(), 21)) + '.txt'
, #BCP varchar(8000)
;
INSERT INTO ##TMP
{COMPLEX SELECT}
;
SET #SQL = 'SELECT * FROM ##TMP;
SET #BCP = 'bcp "' + #sql + '" queryout "' + #path + '" -c -T -S.'
EXEC xp_cmdshell #bcp;
Not nice, but easy and working.
Put the complete bcp query in one not to change the line while writing bcp query.
You can write your query as:
Declare #sql Varchar(8000)
Select #sql = 'bcp "SELECT FirstName, LastName FROM AdventureWorks2012.Person.Person ORDER BY LastName, Firstname " queryout "C:\Users\David\Desktop\yes.txt" -Usa -Ppassw0rd -c -t, -T -S'
EXEC master..xp_cmdshell #sql

Exporting CSV's from SQL Server - Iterating different Key

I am currently going through the process of exporting a CSV file from an SQL Server DB for each unique key in a database that contains meter readings. I am doing this one at a time manually via Export data in SSMS.... It is making me want to die since I have 200 unique key values.
Here is my query:
SELECT DataTime, DataValue
FROM [i96X].[dbo].[PointValue]
WHERE PointID = 68352
The bulk of the query stays the same, the only value i change is the PointID. I need tab separated CSV's as output where the filename = the PointID.
Can someone help?
You can use xp_cmdshell and bcp to achieve what you want but it requires more privileges, in case you're thinking about running this in production.
You need to configure enable advanced options in order to enable xp_cmdshell and then enable xp_cmdshell
EXEC sp_configure 'show advanced options', 1 RECONFIGURE
EXEC sp_configure 'xp_cmdshell', 1 RECONFIGURE
Then we gather all PointIDs into a cursor so we can process individually
Declare curPointIDS Cursor For
Select Distinct PointID From [i96X].[dbo].[PointValue]
Open curPointIDS
Declare #pointId Int, #cmd Varchar(max)
Fetch Next From curPointIDS Into #pointId
While ##FETCH_STATUS = 0
Begin
Set #cmd = 'bcp "SELECT DataTime, DataValue FROM [i96X].[dbo].[PointValue] '
+ 'WHERE PointID = ' + LTRIM(#pointId) + '" queryout '
+ '"C:\Temp\Results_' + LTRIM(#pointId) + '.csv" -T -c -t,'
EXEC xp_cmdshell #cmd
End
Close curPointIDS
Deallocate curPointIDS
xp_cdshell: Runs commands in command prompt (https://msdn.microsoft.com/en-us/library/ms175046.aspx)
bcp: Sql Server's data extract utility https://msdn.microsoft.com/en-us/library/ms162802.aspx
Parameters:
-T: Use trusted connection
-c: Data type will be characters
-t: Specify delimiter, in this case it's comma (,)
You can use a loop and a temp table to accomplish this.
declare #i int = (select count(distinct PointID) from [i96X].[dbo].[PointValue])
select row_number() over(order by PointID) as rn, *
into #tmp --temp table
from [i96X].[dbo].[PointValue]
declare #id int
declare #sql varchar(4000)
while #i > 0
begin
set #id = (select PointID from #tmp where rn = #i)
SELECT DataTime, DataValue
FROM [i96X].[dbo].[PointValue]
WHERE PointID = #id;
select #sql = 'bcp "SELECT DataTime, DataValue FROM [i96X].[dbo].[PointValue] WHERE PointID = '+str(#id)+'" queryout "C:\Temp\'+str(#id)+'+'.csv'+'" -S servername -d databasename -U username -P password'
exec master..xp_cmdshell #sql
set #i = #i-1;
end
If you have a fix list of meters you can create a batch file and extract this information, on a regular basis, using the Task Scheduler.
Basically you should put on this batch file one row for each meter you want to extract and use sqlcmd to extract the historical records.
For example, let's say you want to extract the historical data for the pointIDs (1, 2, 3). You can create a batch file called "extract.bat" with the following lines:
sqlcmd -Q "select * from i96x.dbo.PointValue where Pointid=1" -s "\t" -o "C:\Results\1.csv"
sqlcmd -Q "select * from i96x.dbo.PointValue where Pointid=2" -s "\t" -o "C:\Results\2.csv"
sqlcmd -Q "select * from i96x.dbo.PointValue where Pointid=3" -s "\t" -o "C:\Results\3.csv"
You just need to run this batch file and get your files. This files will use tab as a column separator. If you want to use comma as column separator just change -s , on each row.
You can use excel to make easier to generate this batch file.
Another head up, with that query you will get ALL the values for each pointid. If you are working with meter probably you will be interested in extract only the last month. To do that you will need to adjust the query.

Export xml data using BCP Command in SQL Server

I am trying to export datable data in xml format but,Problem is like i can able to create xml file but data is not get writing in to the file following is my code I am trying.
DECLARE #cmd VARCHAR(2000);
SET #cmd = 'bcp.exe "select * from emp FOR XML AUTO" queryout E:\rahul_1.xml -x -T';
EXEC xp_cmdshell #cmd ;
And following is the output message I am getting after executing above code
NULL
Enter the file storage type of field XML_F52E2B61-18A1-11d1-B105-00805F49916B [ntext]:
can any body please suggest me on this
Dan you answer works, except one last thing. BCP needs additional information about the source query. Best idea to fully qualify the source of the data.
SET #cmd = 'bcp.exe "select * from [Database].[Schema].[Table] FOR XML AUTO"
queryout E:\rahul_1.xml -c -T';
Instead of the -x parameter (generate xml format file), specify -c (character file):
SET #cmd = 'bcp.exe "select * from emp FOR XML AUTO" queryout E:\rahul_1.xml -c -T';
Try to use -w switch for exporting XML file in correct format
DECLARE #cmd VARCHAR(2000);
SET #cmd = 'bcp.exe "select * from [Database].[Schema].[Table] FOR XML AUTO" queryout E:\filename.xml -S MyServer\MyInstance -c -T -w';
EXEC xp_cmdshell #cmd;

Resources