Export data in csv using bat file - sql-server

I'm trying to export my data to a csv file using the command below. I am using -s, for comma delimited.
It's working fine but I face one problem when my customer name has comma(,) inside. In this case it's cutting the customer name in two different column.
How do I resolve this issue?
sqlcmd -S . -d MYDB -E -Q "set nocount on; select 'customer_id','customer_name','salesrep_id'; select customer_id,customer_name,salesrep_id from customer where customer_id=106866" -b -o C:\customer.csv -h-1 -s, -w 700

It would be nice to know your datatypes. But generally put varchar in double quotes. Here's the very basic idea:
select customer_id, '"' + customer_name + '"', salesrep_id from customer where customer_id=106866
I generally like to explicitly create the csv line myself, so my selects usually look more like this:
select '"' + customer_id + '","' + customer_name + '"," + salesrep_id + '"' from customer where customer_id=106866
If there are columns that are numbers, then I convert the column to varchar and don't put double quotes around it.
Based on your comments:
Here is a .cmd file that I use in production to create a .csv file:
set proc=NoBillsReport
sqlcmd -E -S win11.net.davisbrownlaw.com -d ProLaw -Q "exec %proc%" -h -1 -W > "c:\2019\no-bills.csv"
It uses a stored procedure and redirects the output instead of trying to have SQL do the output.
Here is a simplified version of my stored procedure:
--Header row
select '"Matter ID","Client Sort","Matter Description","Billing Initials","Area of Law","Fees No-Billed","Costs No-Billed"'
-- Costs that were no-billed in previous month
select '"' + matterid + '","' + clientsort + '","' + shortdesc + '","'
+ initials + '","' + areaoflaw + '",' + convert(varchar(50),nobilled_fees)
+ ',' + convert(varchar(50),nobilled_costs)
from Matters
And its output:
"Matter ID","Client Sort","Matter Description","Billing Initials","Area of Law","Fees No-Billed","Costs No-Billed"
"70441233","ACME","Closing Wheel, Whenever","LA","IP",0.00,7.98
which works great as the contents of a .csv file.
The stored procedure is nice because it doesn't all have to fit on 1 long command line. But, for your use, I think this has a shot at working:
sqlcmd -S . -d MYDB -E -Q "set nocount on; select '""customer_id"",""customer_name"",""salesrep_id""'; select convert(varchar(50),customer_id) + ',""' + customer_name + '"",""' + salesrep_id + '""' from customer where customer_id=106866; " -h -1 -W > C:\customer.csv
Note that this is called from a .cmd file. The double double quotes might be different if called directly from the command line. Also, on my system, the 11 spaces after the ; at the end of the sql line is necessary due to how the double double quotes are consolidated. And yes, there are single quotes and double double quotes all over the place.
Based on further comments, here is how I script a .cmd file to pull year, month, day:
:: Get current year/month/day
for /f "skip=1 tokens=2-4 delims=(-)" %%a in ('"echo.|date"') do (
for /f "tokens=1-3 delims=/.- " %%A in ("%Date:* =%") do (
set %%a=%%A&set %%b=%%B&set %%c=%%C)
)
)
set /a "yy=10000%yy% %% 10000 %% 2000 + 2000,mm=100%mm% %% 100,dd=100%dd% %% 100"
:: Pad with leading zeros if needed
set mm=0%mm%
set mm=%mm:~-2%
set dd=0%dd%
set dd=%dd:~-2%
Then your filename would be "customer%yy%%mm%%dd%.csv"

Related

How to escape this command in a batch file?

I want to generate dynamically a set of switches for PG_DUMP like below:
--table=mySchema.foo --table=mySchema.bar ...
However, I want to restrict those switches to views only. The views names don't follow a pattern. They all reside in a single schema called mySchema.
Here is the batch file script I wrote:
#echo off
set PARAM_HOTE=localhost
set PARAM_PORT=5435
set PSQL="C:\Program Files\PostgreSQL\9.4\bin\psql.exe"
set SQL_QUERY=^
select string_agg( '--table=' || quote_ident(nspname) || '.' || quote_ident(relname), ' ' )^
from (^
select *^
from pg_class^
join pg_namespace on pg_namespace.oid = pg_class.relnamespace^
where relkind = 'v'^
and nspname = 'mySchema'^
order by relname ASC^
) infos_vues^
;
for /f %%i in ('"%PSQL%" --quiet --tuples-only --host %PARAM_HOTE% --port %PARAM_PORT% --username "rec" -c "%SQL_QUERY%" db') do set PG_DUMP_SWITCHES_FOR_VIEWS_ONLY=%%i
:: Call PG_DUMP...
When I run it, I am getting the following error:
'"C:\Program Files\PostgreSQL\9.4\bin\psql.exe"" -c "select' is not recognized as an internal
or external command, operable program or batch file.
Here is how I solved my issue:
#echo off
set PARAM_HOTE=localhost
set PARAM_PORT=5435
set PSQL="C:\Program Files\PostgreSQL\9.2\bin\psql.exe"
set SQL_LISTE_VUES=^
select string_agg( concat('--table=' , quote_ident(nspname) , '.' , quote_ident(relname)), ' ' )^
from (^
select *^
from pg_class^
join pg_namespace on pg_namespace.oid = pg_class.relnamespace^
where relkind = 'v'^
and nspname = 'rec'^
order by relname ASC^
) infos_vues^
;
for /f "usebackq delims=" %%i in (`%%PSQL%% --quiet --tuples-only --host %PARAM_HOTE% --port %PARAM_PORT% --username "rec" -c "%SQL_LISTE_VUES%" REC`) do set LISTE_VUES=%%i
echo %LISTE_VUES%
I rewrote my query by replacing || with the concat function
I used back ticks
I escaped % with %% in the for command

Bash array variable into single space elements

I'm trying to print a word from variable with multiple elements using array. Here is a script:
location=($(here mysql query to extract name of locations))
for i in "${location[#]}"; do
echo "$i"
done
But these location name contains spaces: MUMBAI - BORIVALI, DELHI - LAJPATNAGAR and so on. So it prints like: Output in debug mode:
+ for i in '"${LOCATION[#]}"'
+ echo DELHI
DELHI
+ for i in '"${LOCATION[#]}"'
+ echo -
-
+ for i in '"${LOCATION[#]}"'
+ echo LAJPATNAGAR
LAJPATNAGAR
+ for i in '"${LOCATION[#]}"'
+ echo MUMBAI
MUMBAI
+ for i in '"${LOCATION[#]}"'
+ echo -
-
+ for i in '"${LOCATION[#]}"'
+ echo BORIVLI
BORIVLI
I've tried with double quotes:
location=("$(here mysql query to extract name of locations)")
then output is
+ for i in '"${LOCATION[#]}"'
+ echo 'DELHI - LAJPATNAGAR
MUMBAI - BORIVLI'
together.
I want output:
+ echo 'DELHI - LAJPATNAGAR'
DELHI - LAJPATNAGAR
+ echo 'MUMBAI - BORIVLI'
MUMBAI - BORIVLI
If the locations don't contain newlines, you can try
while read -r location ; do
echo "$location"
done < <(mysql ...)
read reads line by line if given just one parameter. By default, it reads from stdin, but with process substitution, we redirected output of the mysql command to it.
Try setting IFS to newline-only:
IFS="$(echo)"
location=($(here mysql query to extract name of locations))
for i in "${location[#]}"; do
echo "$i"
done
Example with test.txt:
A B C
D E
F
Run the above code, replacing the MySQL query with cat test.txt. Output:
A B C
D E
F
If you drop the IFS="$(echo)" line, you will get each letter on a separate line.
Just count 3 fields:
location=($(here mysql query to extract name of locations))
for ((i=0; i <= ${#location}; i+=3))
do
echo "${location[i]} ${location[i+1]} ${location[i+2]}"
done

Issue with record length when creating CSV in SQLCMD

I am experiencing an issue when generating a csv file from SQLCMD. I have a test table which looks like this.
Which has 2 numeric columns and 2 nvarchar columns (one 64 characters long and the other 2000 characters long).
If I issue the command select * from test what I get is this
number1 number2 varcharacter1 varcharacter2
1 2 test longertext
2 3 test2 longertext2
3 4 test3 longertext3
If I want to output this as csv I use this sql:
select cast(number1 as nvarchar) + ',' +
varcharacter2 + ',' +
varcharacter1 + ',' +
cast(number2 as nvarchar)
from test
This gives me this which looks OK
(No column name)
1,longertext,test,2
2,longertext2,test2,3
3,longertext3,test3,4
The issue is when I enter it via SQLCMD. If I issue the following command
Scripts>sqlcmd -S localhost\STUDIO -d studiodb -U sa -P Studio2016! -h-1 -i test.sql -o test.csv
What I get is a CSV file which contains the data, but when I open it in TextPad the last column is padded out to 2000 characters long.
1,longertext,test,2
2,longertext2,test2,3
3,longertext3,test3,4
What is causing this and how can I fix it? What am I doing wrong?

DB strange behaviour

I encountered the following confusing problem:
A DB table was updated through CSV file and some KSH script:
#!/bin/ksh
moduleDir=$ACS_INSTALL/FMC_TER_AM_028/Database
logFile=$moduleDir/install_scripts/`basename $0`_$(date '+%Y%m%d_%H%M').log
(
cp -ri $moduleDir/ntscripts/datafiles/* $PROVHOME/database/ntscripts/datafiles
echo "\n***********************"
echo "* New/Modified files: *"
echo "***********************"
find $moduleDir/ntscripts/datafiles -type f | xargs ls -l
) 2>&1 | tee $logFile
echo "\nInstallation of Database completed\n"
The update is equal (under equal I mean that the visual result is the same as the INSERT command below) to the following INSERT query:
INSERT INTO tp_nt_mapping (TARIFF_PLAN, SERVICE_TYPE, NETWORK_TEMPLATE, IN_CREATE, IN_DELETE)
VALUES (171, 'Postpaid', '11', '', '')
When I'm using the following SELECT command:
SELECT * FROM tp_nt_mapping ORDER BY tariff_plan DESC
I'm able to see the new inserted record, but when I try with any of the following SELECT queries, I'm not:
SELECT * FROM tp_nt_mapping WHERE network_template = 11 ORDER BY tariff_plan DESC
SELECT * FROM tp_nt_mapping WHERE network_template = '11' ORDER BY tariff_plan DESC
Any suggestions?
The value in network_template field isn't just '11' but '11' || chr(13).
So you have the carrige return char at the end.
You can fix the data by doing:
update tp_nt_mapping
set network_template = replace(network_template, chr(13), '')
But better check why it was added on the first place....

How to download Postgres bytea column as file

Currently, i have a number of files stored in postgres 8.4 as bytea. The file types are .doc, .odt, .pdf, .txt and etc.
May i know how to download all the file stored in Postgres because i need to to do a backup.
I need them in their original file type instead of bytea format.
Thanks!
One simple option is to use COPY command with encode to hex format and then apply xxd shell command (with -p continuous hexdump style switch). For example let's say I have jpg image in bytea column in samples table:
\copy (SELECT encode(file, 'hex') FROM samples LIMIT 1) TO
'/home/grzegorz/Desktop/image.hex'
$ xxd -p -r image.hex > image.jpg
As I checked it works in practice.
Try this:
COPY (SELECT yourbyteacolumn FROM yourtable WHERE <add your clauses here> ...) TO 'youroutputfile' (FORMAT binary)
Here's the simplest thing I could come up with:
psql -qAt "select encode(file,'base64') from files limit 1" | base64 -d
The -qAt is important as it strips off any formatting of the output. These options are available inside the psql shell, too.
base64
psql -Aqt -c "SELECT encode(content, 'base64') FROM ..." | base64 -d > file
xxd
psql -Aqt -c "SELECT encode(content, 'hex') FROM ..." | xxd -p -r > file
If you have a lot of data to download then you can get the lines first and then iterate through each one writing the bytea field to file.
$resource = pg_connect('host=localhost port=5432 dbname=website user=super password=************');
// grab all the user IDs
$userResponse = pg_query('select distinct(r.id) from resource r
join connection c on r.id = c.resource_id_from
join resource rfile on c.resource_id_to = rfile.id and rfile.resource_type_id = 10
join file f on rfile.id = f.resource_id
join file_type ft on f.file_type_id = ft.id
where r.resource_type_id = 38');
// need to work through one by one to handle data
while($user = pg_fetch_array($userResponse)){
$user_id = $user['id'];
$query = 'select r.id, f.data, rfile.resource_type_id, ft.extension from resource r
join connection c on r.id = c.resource_id_from
join resource rfile on c.resource_id_to = rfile.id and rfile.resource_type_id = 10
join file f on rfile.id = f.resource_id
join file_type ft on f.file_type_id = ft.id
where r.resource_type_id = 38 and r.id = ' . $user_id;
$fileResponse = pg_query($query);
$fileData = pg_fetch_array($fileResponse);
$data = pg_unescape_bytea($fileData['data']);
$extension = $fileData['extension'];
$fileId = $fileData['id'];
$filename = $fileId . '.' . $extension;
$fileHandle = fopen($filename, 'w');
fwrite($fileHandle, $data);
fclose($fileHandle);
}
DO $$
DECLARE
l_lob_id OID;
r record; BEGIN
for r in
select data, filename from bytea_table
LOOP
l_lob_id:=lo_from_bytea(0,r.data);
PERFORM lo_export(l_lob_id,'/home/...'||r.filename);
PERFORM lo_unlink(l_lob_id);
END LOOP;
END; $$
Best I'm aware, bytea to file needs to be done at the app level.
(9.1 might change this with the filesystem data wrapper contrib. There's also a lo_export function, but it is not applicable here.)
If you want to do this from a local windows, and not from the server, you will have to run every statement individually, and have PGAdmin and certutil:
Have PGAdmin installed.
Open cmd from the runtime folder or cd "C:\Program Files\pgAdmin 4\v6\runtime"
Run in PGAdmin query to get every statement that you will have to paste in cmd:
SELECT 'set PGPASSWORD={PASSWORD} && psql -h {host} -U {user} -d {db name} -Aqt -c "SELECT encode({bytea_column}, ''base64'') FROM {table} WHERE id='||id||'" > %a% && CERTUTIL -decode %a% "C:\temp{name_of_the_folder}\FileName - '||{file_name}||' ('||TO_CHAR(current_timestamp(),'DD.MM.YYYY,HH24 MI SS')||').'||{file_extension}||'"'
FROM table WHERE ....;
Replace {...}
It will generate something like:
set PGPASSWORD=123 psql -h 192.1.1.1 -U postgres -d my_test_db -Aqt -c "SELECT encode(file_bytea, 'base64') FROM test_table_bytea WHERE id=33" > %a% && CERTUTIL -decode %a% "C:\temp\DB_FILE\FileName - test1 - (06.04.2022,15 42 26).docx"
set PGPASSWORD=123 psql -h 192.1.1.1 -U postgres -d my_test_db -Aqt -c "SELECT encode(file_bytea, 'base64') FROM test_table_bytea WHERE id=44" > %a% && CERTUTIL -decode %a% "C:\temp\DB_FILE\FileName - test2 - (06.04.2022,15 42 26).pdf"
Copy paste all the generated statements in CMD. The files will be saved to your local machine.

Resources