T-SQL procedure - BCP - export CSV table with header - sql-server

I have run the following query to export my SQL Server table as CSV. It is working fine. But now I want to add the column names as the first row. How is that possible?
DECLARE #archivoOUT varchar(800)
DECLARE #sql nvarchar(1000)
SET #archivoOUT = CONCAT('D:\archivosolicitudrestcate', FORMAT (GETDATE(), 'yyyyMMdd'),'.csv')
SET #sql = 'bcp "[dbo].[TEMP_res]" out '+#archivoOUT+' -S '+##SERVERNAME+' -d CentroMedico -c -T -w'
EXEC master..xp_cmdshell #sql

To add column names to your BCP out, you can change your syntax slightly.
You will need to select the columns that you want from the table instead of BCP'ing the entire table.
Currently you have,
'bcp "[dbo].[TEMP_res]" out '...
Modify the query syntax slightly. To select specific columns from the table try,
'bcp "select 'column1', 'column2'
union all
SELECT column1, column2 FROM [testdb].[dbo].[TEMP_res]" out' ...
More details at Microsoft's learning site, but here is an even better answer directly from StackOverflow.

The method I always relied is the one referenced in the link #GuiLeFlea mentioned where you concatenate column and detail rows separately.
DECLARE #archivoOUT varchar(800)
DECLARE #archivoOUTdetails varchar(800)
DECLARE #sql nvarchar(1000)
SET #archivoOUT = CONCAT('D:\archivosolicitudrestcate', FORMAT (GETDATE(), 'yyyyMMdd'),'.csv')
SET #archivoOUTdetails = CONCAT('D:\archivosolicitudrestcate', FORMAT (GETDATE(), 'yyyyMMdd'),'_details.csv')
SET #sql = 'bcp "select ''column1'', ''column2'', ''column3''" queryout '+#archivoOUT+' -S '+##SERVERNAME+' -d CentroMedico -c -T -w'
EXEC master..xp_cmdshell #sql
SET #sql = 'bcp "[dbo].[TEMP_res]" out '+#archivoOUTdetails+' -S '+##SERVERNAME+' -d CentroMedico -c -T -w'
EXEC master..xp_cmdshell #sql
SET #sql = 'cmd /U /C type ' + #archivoOUTdetails + ' >> ' + #archivoOUT
EXEC master..xp_cmdshell #sql
The advantage is this will always order by correctly, regardless of execution plan.

So another way you can achieve your goal, guarantee the rows are ordered, and do it in a simple manner that only requires 1 call to xp_cmdshell is by adding a dummy sort ID column to the UNION ALL query, and then wrapping it in a CTE or subquery so you can order on it without having to select it:
DECLARE #archivoOUT varchar(800)
DECLARE #sql nvarchar(1000)
SET #archivoOUT = CONCAT('D:\archivosolicitudrestcate', FORMAT (GETDATE(), 'yyyyMMdd'),'.csv')
SET #sql =
CONCAT
(
N'bcp ',
N'"SELECT Column1, Column2, Column3 ',
N'FROM ',
N'( ',
N' SELECT ''Column1Name'' AS Column1, ''Column2Name'' AS Column2, ''Column3Name'' AS Column3, 1 AS SortId',
N' UNION ALL ',
N' SELECT Column1, Column2, Column3, 2 AS SortId ',
N' FROM dbo.TEMP_res ',
N') AS Results ',
N'ORDER BY SortId" ',
N'out ', #archivoOUT, ' -S ', ##SERVERNAME, ' -d CentroMedico -c -T -w'
)
EXEC master..xp_cmdshell #sql
This is kind of the best of both worlds and then some, from the other answers.
Note I'm using CONCAT() so I can format the query in a human readable way, that's just my preference, it's not required. You can just stuff the whole code in a single line of code if you prefer, like your original BCP query string.

I just create a view that does this:
SELECT 'Field_1, Field_2, Field_3'
union all
SELECT Field_1,Field_2,Field_3
FROM Table
Edit: A UNION ALL does guarantee the datasets will be in order. The rows in each dataset may not be, but the order of the outputted datasets will be in the order they are executed. Just look at the Execution Plan, it always ends with "Concatenation - Append multiple input tables to form the output table"

Related

BCP - Export CSV with header

I have run the following query to export my Ms SQL table as CSV. It working good. Now I want to add the field name as the first row. How is it possible?
declare #sql varchar(8000)
select #sql = 'bcp "select * from test_table" queryout C:\Test_SP\Tom.csv -c -t, -T -S' + ##servername
exec master..xp_cmdshell #sql
I know that I can specify the names #Red Devil answered. But the table is dynamic, Its fields are not fixed, It will change. I am trying to find a method to fetch the field names from the table definition and prepend it into the result CSV
Try this:
declare #sql varchar(8000)
select #sql = 'bcp "select 'col1', 'col2',... union all select * from test_table" queryout C:\Test_SP\Tom.csv -c -t, -T -S' + ##servername
exec master..xp_cmdshell #sql

Dynamic file creation with BCP Utility

I'm using BCP Utility to copy records out of table before deleting the records.
The function is working just fine, however, I need to copy the records to a new file for the every time I delete, instead of override the same file (as it is now).
It could be creating a new file with timestamp as prefix or something similar.
Any ideas?
My code
Declare #cmd varchar(1000) = 'bcp "select * from ##DeletedRecords" queryout
"C:\Delete\DeletedRecord.txt" -t, -c -T'
print #cmd
EXEC master..XP_CMDSHELL #cmd
just change the filename in the BCP command accordingly by appending date & time to the filename
example :
Declare #cmd varchar(1000);
select #cmd = 'bcp "select * from ##DeletedRecords" queryout '
+ '"C:\Delete\DeletedRecord'
+ convert(varchar(10), getdate(), 112) -- YYYYMMDD
+ replace(convert(varchar(10), getdate(), 108), ':', '') -- HHMMSS
+ '.txt" -t, -c -T'
print #cmd
EXEC master..XP_CMDSHELL #cmd

SQL Server User Defined Table Types and BCP Export

I can't seem to get this script to work. I'm getting the following error:
Msg 137, Level 16, State 1, Line 14
Must declare the scalar variable "#TVP_GLICU".
Can anyone tell me what am I missing?
Declare #TVP_GLICU TVP_GLICU
DECLARE #cmd varchar(500)
Declare #TimeStamp as nvarchar(100) = Replace((CONVERT(varchar(25), getdate(), 121)),':','')
--Insert Batch numbers in user defined table types
Insert Into #TVP_GLICU (ID)
Values ('563704')
Insert Into #TVP_GLICU (ID)
Values ('498721')
--select *
--From #TVP_GLICU
SET #cmd = 'BCP "EXECUTE [F0902].[D365O].[Get-F0911NewRecords]'+#TVP_GLICU+'" QUERYOUT "D:\D365O\DataSource\F0911\'+#TimeStamp+'.csv" -c -t\^, -T -S' + ##SERVERNAME + ''
EXECUTE MASTER..xp_cmdshell #cmd
You can't refer to a table variable from BCP that was created outside the scope of the query parameter. When BCP is executed, it creates a new session and the scope of table variables is limited to the session that created the table variable.
Problem is that you are concatenating string and are trying to append a table-valued type:
SET #cmd = 'BCP "EXECUTE [F0902].[D365O].[Get-F0911NewRecords]'+#TVP_GLICU+'" QUERYOUT "D:\D365O\DataSource\F0911\'+#TimeStamp+'.csv" -c -t\^, -T -S' + ##SERVERNAME + ''
Can you instead create a Staging table in the database and use its name in BCP query?

Why do I get an error in bcp query out where clause?

I'm new to SQL Server and write this query for save select result into csv file:
declare #Cycle_ID as int
set #Cycle_ID = 0
EXECUTE master.dbo.xp_cmdshell 'bcp "select [Telno],[Cycle],[Price] FROM [ClubEatc].[dbo].[CycleAnalysisTable] where cast([Price] as float)>'+ #Cycle_ID +' " queryout d:\download\behi.csv -t"|" -c -S VM_TAZMINDARAMA -U behzad -P beh1368421'
In where clause I write simple variable, but I get this error:
Incorrect syntax near '+'.
Please don't decrease my question! I'm new! Thanks
SQL Server doesn't recognize expressions in exec statements. So, try setting up the query first in a variable and using that:
declare #Cycle_ID as int;
set #Cycle_ID = 0;
declare #sql nvarchar(max);
set #sql = '
bcp "select [Telno],[Cycle],[Price] FROM [ClubEatc].[dbo].[CycleAnalysisTable] where cast([Price] as float)>'+ cast(#Cycle_ID as varchar(255)) +' ";
EXECUTE master.dbo.xp_cmdshell #sql queryout d:\download\behi.csv -t"|" -c -S VM_TAZMINDARAMA -U behzad -P beh1368421';
It seems curious to me that you are comparing a column called Price to a variable called #Cycle_ID, but that has nothing to do with the syntax issue.

Exporting CSV's from SQL Server - Iterating different Key

I am currently going through the process of exporting a CSV file from an SQL Server DB for each unique key in a database that contains meter readings. I am doing this one at a time manually via Export data in SSMS.... It is making me want to die since I have 200 unique key values.
Here is my query:
SELECT DataTime, DataValue
FROM [i96X].[dbo].[PointValue]
WHERE PointID = 68352
The bulk of the query stays the same, the only value i change is the PointID. I need tab separated CSV's as output where the filename = the PointID.
Can someone help?
You can use xp_cmdshell and bcp to achieve what you want but it requires more privileges, in case you're thinking about running this in production.
You need to configure enable advanced options in order to enable xp_cmdshell and then enable xp_cmdshell
EXEC sp_configure 'show advanced options', 1 RECONFIGURE
EXEC sp_configure 'xp_cmdshell', 1 RECONFIGURE
Then we gather all PointIDs into a cursor so we can process individually
Declare curPointIDS Cursor For
Select Distinct PointID From [i96X].[dbo].[PointValue]
Open curPointIDS
Declare #pointId Int, #cmd Varchar(max)
Fetch Next From curPointIDS Into #pointId
While ##FETCH_STATUS = 0
Begin
Set #cmd = 'bcp "SELECT DataTime, DataValue FROM [i96X].[dbo].[PointValue] '
+ 'WHERE PointID = ' + LTRIM(#pointId) + '" queryout '
+ '"C:\Temp\Results_' + LTRIM(#pointId) + '.csv" -T -c -t,'
EXEC xp_cmdshell #cmd
End
Close curPointIDS
Deallocate curPointIDS
xp_cdshell: Runs commands in command prompt (https://msdn.microsoft.com/en-us/library/ms175046.aspx)
bcp: Sql Server's data extract utility https://msdn.microsoft.com/en-us/library/ms162802.aspx
Parameters:
-T: Use trusted connection
-c: Data type will be characters
-t: Specify delimiter, in this case it's comma (,)
You can use a loop and a temp table to accomplish this.
declare #i int = (select count(distinct PointID) from [i96X].[dbo].[PointValue])
select row_number() over(order by PointID) as rn, *
into #tmp --temp table
from [i96X].[dbo].[PointValue]
declare #id int
declare #sql varchar(4000)
while #i > 0
begin
set #id = (select PointID from #tmp where rn = #i)
SELECT DataTime, DataValue
FROM [i96X].[dbo].[PointValue]
WHERE PointID = #id;
select #sql = 'bcp "SELECT DataTime, DataValue FROM [i96X].[dbo].[PointValue] WHERE PointID = '+str(#id)+'" queryout "C:\Temp\'+str(#id)+'+'.csv'+'" -S servername -d databasename -U username -P password'
exec master..xp_cmdshell #sql
set #i = #i-1;
end
If you have a fix list of meters you can create a batch file and extract this information, on a regular basis, using the Task Scheduler.
Basically you should put on this batch file one row for each meter you want to extract and use sqlcmd to extract the historical records.
For example, let's say you want to extract the historical data for the pointIDs (1, 2, 3). You can create a batch file called "extract.bat" with the following lines:
sqlcmd -Q "select * from i96x.dbo.PointValue where Pointid=1" -s "\t" -o "C:\Results\1.csv"
sqlcmd -Q "select * from i96x.dbo.PointValue where Pointid=2" -s "\t" -o "C:\Results\2.csv"
sqlcmd -Q "select * from i96x.dbo.PointValue where Pointid=3" -s "\t" -o "C:\Results\3.csv"
You just need to run this batch file and get your files. This files will use tab as a column separator. If you want to use comma as column separator just change -s , on each row.
You can use excel to make easier to generate this batch file.
Another head up, with that query you will get ALL the values for each pointid. If you are working with meter probably you will be interested in extract only the last month. To do that you will need to adjust the query.

Resources