BCP Command to Export Query Results to File with Pipe Delimiter - sql-server

I have a requirement to create a SQL Job that exports a query to a Pipe / Vertical bar delimited file (|) and save it on a network drive in either *.txt or *.csv format. Right now, I am just trying to get this to work inside SSMS, calling the BCP command and exporting the stored procedure in the proper format to the network location, but not able to get it working.
I have been researching this and there are two methods for this.
Use the export data wizard to create a job and schedule that to run. But this method, if we need to make changes, I believe we cannot change the SSIS package that is created so we lose flexibility
Use the BCP command to export the file.
I greatly prefer to use option #2, the BCP command, but I am having problems. I just cannot seem to get the syntax correct and hoping someone could show me what I am missing:
This is my command:
Exec master..xp_cmdshell 'bcp EXEC [dbo].[usp_Report_1123] ''7786'' -t| out \\networkDrive\Reports\REPORT_1123\report1123.csv -T'
But I get the following messages:
output
'out' is not recognized as an internal or external command,
operable program or batch file.
NULL
The stored procedure does work and returns data. The network path, if I enter it into my computer, finds the path. But I am not sure what I am missing and hoping someone could help.

Related

Export data with bcp utility with double quotes

I'm trying to exporting data from MS sql server by using bcp utility command line. The problem is that in the exported output is missing the first double quote at the first line and I cannot explain the reason.
Below the command that I'm using for the export:
/opt/mssql-tools/bin/bcp db_schema out dump.csv -c -t"\",\"" -r"\"\n\"" -S my_host -U my_user
But the output result is missing the first double quotes on first line (only the first line) of the exported csv file:
801","40116","Hazelnut MT -L","Thursday Promo","Large","","5.9000","","801","1.0000","","3.6500","2.2500",".0000","default","","","","","Chatime","02/06/2014","09125a9cfffd4143a00e73e3b62f15f2","CB01","",".0000","5.9000","6.9000",".0000",".0000",".0000",".0000",".0000",".0000","0","","0","0","0","","","","","","","","","Modern Milk Tea","","","0","","","1","0","","","","","","","","0","Hau Chan","","","","","","","","","","0","","","","","","","-1","","","","","","","","","","","","0","00000000420714AA","2014-06-02","1900-01-01","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","",""
Am I missing something?

If you know the field names, you can try the following:
Create Table [names]
(
Id Int,
fname VarChar(15),
lname VarChar(25)
)
Insert Into names Values
(1, 'Jim', 'Morrison'),
(2,'Robert', 'Plant'),
(3,'Janis', 'Joplin')
BCP command: Using quotename() with char(34)
(This BCP command uses a Trusted connection)
bcp "SELECT quotename(Cast(Id As VarChar(15)),char(34)), quotename(fname,char(34)), quotename(lname,char(34)) FROM names" queryout dump.csv -c -t"," -S SERVER -d DBNAME -T
Result:
"1","Jim","Morrison"
"2","Robert","Plant"
"3","Janis","Joplin"

I'll bet that you also have a line at the end of the file that is just a single double-quote. Why? With the command line switches you've provided, you're saying "end every field with "," and every row with "\n"". And that's what it's doing.
So, the second and subsequent lines start with a double-quote because the previous line ends with one.
Clippy DBA says "it looks like you're trying to produce a CSV". Without knowing why, it's hard for me... er... Clippy to suggest an alternative. Specifically, what is going to be reading this file? Will you be reading this with Excel or something else that's expecting a specific format? WIll you be importing it into another database table?

The quotename answer provided by level3looper will work in your case.
For completeness, Im providing a solution I've given in the past for the same purpose.
I like this one a little better because it keeps the definition of formatting in the format file, which is where I prefer to go to get that info. The quotename is a good solution for quick, adhoc work, but for automation, business process, I would recommand the link below.
Essentially, you just add a "dummy" column to the beginning of the definition of the extract file and delimit that column with a single doublequote. Then you also note in the format file to skip the first column. This gives you just the doublequote at the start of the line.
sql server bcp bulk insert pipe delimited with text qualifier format file

How to import file contents into table column as data

A project I'm working on at work involves modifying one of the subsystems to store/pull data that is currently stored in files into the database. Each of the files is a single, sometimes-large, chunk of custom (xml-based) script generated by another custom tool.
Conceptually, I'm looking for an easy way to do something like:
For Each file in folder_and_subfolders
INSERT INTO table
(script_name, version_num, script )
VALUES
({file_name}, 1, {file_contents})
;
Next
Preferably on an entire directory tree at once.
If there's no easy way to do this via T-SQL, I can write a utility to do the job, but I'd prefer something that didn't require having to write another custom tool that will only be used once.

So, I don't have SQL Server installed and therefore can't test this, but if you are looking for a simple batch file that could do what you're after, I'd suggest something like the following might well help;
#echo off
SET xmldir=./myxmlfiles/live/here/
echo --- Processing files
for %%f in ("%xmldir%*.xml") do (echo Running %%f.... && #sqlcmd -I -U %1 -P %2 -S %3 -d %4 -v filename="%xmldir%%%f" -i ProcessFile.sql)
I'm not sure how much you know about sqlcmd, but it is a command line tool that is generally provided by SQL Server. It will allow you to run SQL commands, or in the case above, run a script which is indicated by the -i parameter. I am assuming that you'd place your SQL statement in there to perform your additions to the table.
The other parameters to sqlcmd are described below;
-I sets QUOTED_IDENTIFIER on (you may or may not need this. I did for an earlier issue I faced with sqlcmd and QUOTED_IDENTIFIER)
-U sets the database username
-P sets the database password
-S sets the database server
-d sets the database to connect to
-v is the interesting one here as it lets you pass parameters to your script. Note that on the MSDN page describing this, it states that if your path or filename contains spaces, then you'll need to enclose it in quotes, so check that out. Basically though, you'd be able to refer to the parameter inside your sql script (ProcessFile.sql) like INSERT INTO mytable (file_name) VALUES ('$(filename)')
You'd have to use the logic described in the answer from my previous comment to ensure

xp_cmdshell command not executing last command when run as job

First off, before everybody shouts at me - I'm bug fixing in legacy code and a re-write is off the cards for now - I have to try to find a fix using the xp_cmdshell command.
I have a proc which is executed via a scheduled job. The proc is full of TSQL like the below to dump data to a log file.
SELECT *
INTO Temp
FROM MyView
SET #cmd1 = 'bcp "SELECT * FROM [myDatabase].dbo.Temp" queryout "C:\temp.txt" -T -c -t" "'
SET #cmd2= 'type "C:\temp.txt" >> "C:\output.txt"'
EXEC master..xp_cmdshell #cmd1
EXEC master..xp_cmdshell #cmd2
DROP TABLE Temp
The problem is that the last of these commands in the proc doesn't appear to run. I can see the result in the text.txt file, but not the output.txt file. All of the preceding work fine though and it works fine when I run this on it's own.
Can anyone suggest why this might happen or suggest an alternative way to achieve this?
Thanks

I think, that BCP as external process runs async. So it could be, that your file is not yet written in the moment you are trying to copy its content.
Suggestion 1: Include an appropriate wait time
Suggestion 2: Call your first command a second time with changed target file name
Suggestion 3: Use copy rather than type
You might create a file c\temp.txt with just hello world in it. Try to type it into one file before the BCP and type it into another file after the BCP.

create flat file in ssis package

I'm working on creating a csv export from a SQL Server database and I've been familiar with a process for doing so that admittedly, I've never completely understood. The process involves creating a "template" file, which defines the columns and structure for the file export. Once the "template" file exists, you can use a Data Flow task to fill it and a File System Task to copy it to the final storage destination with whatever file name you'd like (frequently a date/time stamp).
Is there a reason that you can't simply create a file directly, without the intermediate "template" file? I've looked around for a bit and it seems like all the proposed solutions involve connecting to an existing file. I see that there is a "Create File" Usage type for a "File" connection manager, but you can't use it in any File System Task. The only File System Type connection managers you can use relative to a file are "Copy", "Delete", "Move", "Rename", and "Set Attributes".
Is there a way to create a file at package run time and fill it?

The whole point of SSIS is to create a data flow with metadata so that the data can be manipulated - if you just want to go database direct to CSV you are probably better off using bcp (bulk copy program) from the command line. If you want to include it as part of a SSIS package just add an Execute Process Task and add the command line to that. You can dynamically change the included columns or the output file by adding an expression to the task. You could also call bcp though TSQL using an Excute SQL Task.
One other option is to concatenate all your columns in your query inter-spaced with a comma literal and output to a text file with just one very wide column.
For documentation on bcp look here

Execute stored procedure by passing the Script File(.sql) as Parameter

I have a stored procedure, in wWhich I m passing the script file (.sql file) as a parameter.
I want to know how .sql file gets executed through command (not command prompt).
exec my_sp VersionNumber, SqlDeltaScript.sql (it is a file)
I want my stored procedure to execute SqlDeltaScript.sql
Can anyone please help regarding this ...
Thanks in advance ...

This does not sound like an ideal situation, but if you have to do this then you could use xp_cmdshell to run the sqlcmd utility to run your script file.
The xp_cmdshell SP must be enabled in order ot use it - see Enable 'xp_cmdshell' SQL Server.

Resources