sqlcmd: reconciling -W and -Y - sql-server

I have an automated batch process that uses sqlcmd to pull data from a database and dump it into a text file. Many fields in that database are of type varchar(max), Sqlcmd limits these fields to 256 characters unless I add something like -y 0 to the flags in the sqlcmd call.
This gives me the full text for fields larger than 256 characters, but it also adds a great deal of whitespace; the fields are padded to make each field as big as it could possibly be according to its data type, essentially giving me huge files with lots of padding and wasted space.
I could fix this by adding -W to my sqlcmd flags, but this gives me an error saying that -W and -y are incompatible.
Has anyone had this problem before? Thoughts on how to solve it?


From this thread there is the suggestion that specifying the column separator using -s can trim the data as if it is not specified the data comes out fixed width.
If that does not work have you tried RTRIM(LTRIM(ColumnName)) in your SELECT query?


I had this issue while creating a CSV file with SQLCMD and got it solved by tricking SQLCMD. Instead of returning several fields and let SQLCMD put the comma as separator, I just concatenate all the fields putting the commas myself in it. I know it's an ugly workaround, but that solved my problem. Hope it can help someone else.

Related

BCP Command to Export Query Results to File with Pipe Delimiter

I have a requirement to create a SQL Job that exports a query to a Pipe / Vertical bar delimited file (|) and save it on a network drive in either *.txt or *.csv format. Right now, I am just trying to get this to work inside SSMS, calling the BCP command and exporting the stored procedure in the proper format to the network location, but not able to get it working.
I have been researching this and there are two methods for this.
Use the export data wizard to create a job and schedule that to run. But this method, if we need to make changes, I believe we cannot change the SSIS package that is created so we lose flexibility
Use the BCP command to export the file.
I greatly prefer to use option #2, the BCP command, but I am having problems. I just cannot seem to get the syntax correct and hoping someone could show me what I am missing:
This is my command:
Exec master..xp_cmdshell 'bcp EXEC [dbo].[usp_Report_1123] ''7786'' -t| out \\networkDrive\Reports\REPORT_1123\report1123.csv -T'
But I get the following messages:
output
'out' is not recognized as an internal or external command,
operable program or batch file.
NULL
The stored procedure does work and returns data. The network path, if I enter it into my computer, finds the path. But I am not sure what I am missing and hoping someone could help.

Export data with bcp utility with double quotes

I'm trying to exporting data from MS sql server by using bcp utility command line. The problem is that in the exported output is missing the first double quote at the first line and I cannot explain the reason.
Below the command that I'm using for the export:
/opt/mssql-tools/bin/bcp db_schema out dump.csv -c -t"\",\"" -r"\"\n\"" -S my_host -U my_user
But the output result is missing the first double quotes on first line (only the first line) of the exported csv file:
801","40116","Hazelnut MT -L","Thursday Promo","Large","","5.9000","","801","1.0000","","3.6500","2.2500",".0000","default","","","","","Chatime","02/06/2014","09125a9cfffd4143a00e73e3b62f15f2","CB01","",".0000","5.9000","6.9000",".0000",".0000",".0000",".0000",".0000",".0000","0","","0","0","0","","","","","","","","","Modern Milk Tea","","","0","","","1","0","","","","","","","","0","Hau Chan","","","","","","","","","","0","","","","","","","-1","","","","","","","","","","","","0","00000000420714AA","2014-06-02","1900-01-01","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","",""
Am I missing something?

If you know the field names, you can try the following:
Create Table [names]
(
Id Int,
fname VarChar(15),
lname VarChar(25)
)
Insert Into names Values
(1, 'Jim', 'Morrison'),
(2,'Robert', 'Plant'),
(3,'Janis', 'Joplin')
BCP command: Using quotename() with char(34)
(This BCP command uses a Trusted connection)
bcp "SELECT quotename(Cast(Id As VarChar(15)),char(34)), quotename(fname,char(34)), quotename(lname,char(34)) FROM names" queryout dump.csv -c -t"," -S SERVER -d DBNAME -T
Result:
"1","Jim","Morrison"
"2","Robert","Plant"
"3","Janis","Joplin"

I'll bet that you also have a line at the end of the file that is just a single double-quote. Why? With the command line switches you've provided, you're saying "end every field with "," and every row with "\n"". And that's what it's doing.
So, the second and subsequent lines start with a double-quote because the previous line ends with one.
Clippy DBA says "it looks like you're trying to produce a CSV". Without knowing why, it's hard for me... er... Clippy to suggest an alternative. Specifically, what is going to be reading this file? Will you be reading this with Excel or something else that's expecting a specific format? WIll you be importing it into another database table?

The quotename answer provided by level3looper will work in your case.
For completeness, Im providing a solution I've given in the past for the same purpose.
I like this one a little better because it keeps the definition of formatting in the format file, which is where I prefer to go to get that info. The quotename is a good solution for quick, adhoc work, but for automation, business process, I would recommand the link below.
Essentially, you just add a "dummy" column to the beginning of the definition of the extract file and delimit that column with a single doublequote. Then you also note in the format file to skip the first column. This gives you just the doublequote at the start of the line.
sql server bcp bulk insert pipe delimited with text qualifier format file

How to import file contents into table column as data

A project I'm working on at work involves modifying one of the subsystems to store/pull data that is currently stored in files into the database. Each of the files is a single, sometimes-large, chunk of custom (xml-based) script generated by another custom tool.
Conceptually, I'm looking for an easy way to do something like:
For Each file in folder_and_subfolders
INSERT INTO table
(script_name, version_num, script )
VALUES
({file_name}, 1, {file_contents})
;
Next
Preferably on an entire directory tree at once.
If there's no easy way to do this via T-SQL, I can write a utility to do the job, but I'd prefer something that didn't require having to write another custom tool that will only be used once.

So, I don't have SQL Server installed and therefore can't test this, but if you are looking for a simple batch file that could do what you're after, I'd suggest something like the following might well help;
#echo off
SET xmldir=./myxmlfiles/live/here/
echo --- Processing files
for %%f in ("%xmldir%*.xml") do (echo Running %%f.... && #sqlcmd -I -U %1 -P %2 -S %3 -d %4 -v filename="%xmldir%%%f" -i ProcessFile.sql)
I'm not sure how much you know about sqlcmd, but it is a command line tool that is generally provided by SQL Server. It will allow you to run SQL commands, or in the case above, run a script which is indicated by the -i parameter. I am assuming that you'd place your SQL statement in there to perform your additions to the table.
The other parameters to sqlcmd are described below;
-I sets QUOTED_IDENTIFIER on (you may or may not need this. I did for an earlier issue I faced with sqlcmd and QUOTED_IDENTIFIER)
-U sets the database username
-P sets the database password
-S sets the database server
-d sets the database to connect to
-v is the interesting one here as it lets you pass parameters to your script. Note that on the MSDN page describing this, it states that if your path or filename contains spaces, then you'll need to enclose it in quotes, so check that out. Basically though, you'd be able to refer to the parameter inside your sql script (ProcessFile.sql) like INSERT INTO mytable (file_name) VALUES ('$(filename)')
You'd have to use the logic described in the answer from my previous comment to ensure

curl output file replacing path seperator with underscore when writing the file

I'm trying to create a fairly simple batch file to download a file from our ftp site and store it in a specific directory. I would like to be able to put the batch file in the path so I can call it from anywhere. Currently I have a line like the following:
curl -v -u %FTP_USER%:%FTP_PASS% -Q "TYPE I" -o %OUTPUT_PATH% "ftp://%FTP_HOST%/%JAR_FILE_NAME%"
What I've discovered is that no matter the value of the output path, the file is always written in the current directory, and any path seperators are converted to underscores in the file name. This happens no matter if I use a '\' or '/' and if I try to escape it with double slashes I just end up with two underscores. Quotes around things don't seem to help either.
My question is does the -o option allow for outputting to a folder other than current working dir? I guess I can have the next step in the script to be "move the file to its destination", but that seems really kludgey.

This sounds exactly like a regression (reported here) we unfortunately brought in curl 7.47.0. We hope to release an updated, fixed, version soon and in the mean time you can probably consider downgrading to an earlier version to work-around this annoying issue.

Exporting UTF8 data from db2

I have a db2 table that contains values in many languages (including right-to-left languages.) When I export this table on a linux box using cli's ''export'' command, I get a good looking comma delimited text file (DEL file,) but when I try it on aix, it replaces all characters that are not in ascii with 0x1a.
I tried playing around with LC_LANG and DB2CODEPAGE, no go. I also tried using codepage modifier, but cli said it can't convert between these two codepages (any codepage I tried that is not English.)
I also tried IXF export, and the data is corrupted there as well.
Help! F1!
Thanks

The codepage of the database has to be set when creating the database. It is not possible to modify it later. You can check the codepage of the database with the following command and look for the value of "Database code page":
db2 get db cfg for [database_name]
Newer AIX versions shouldn't have problems with Unicode, but if you have and older version, that might cause problems too.

Resources