I want to generate dynamically a set of switches for PG_DUMP like below:
--table=mySchema.foo --table=mySchema.bar ...
However, I want to restrict those switches to views only. The views names don't follow a pattern. They all reside in a single schema called mySchema.
Here is the batch file script I wrote:
#echo off
set PARAM_HOTE=localhost
set PARAM_PORT=5435
set PSQL="C:\Program Files\PostgreSQL\9.4\bin\psql.exe"
set SQL_QUERY=^
select string_agg( '--table=' || quote_ident(nspname) || '.' || quote_ident(relname), ' ' )^
from (^
select *^
from pg_class^
join pg_namespace on pg_namespace.oid = pg_class.relnamespace^
where relkind = 'v'^
and nspname = 'mySchema'^
order by relname ASC^
) infos_vues^
;
for /f %%i in ('"%PSQL%" --quiet --tuples-only --host %PARAM_HOTE% --port %PARAM_PORT% --username "rec" -c "%SQL_QUERY%" db') do set PG_DUMP_SWITCHES_FOR_VIEWS_ONLY=%%i
:: Call PG_DUMP...
When I run it, I am getting the following error:
'"C:\Program Files\PostgreSQL\9.4\bin\psql.exe"" -c "select' is not recognized as an internal
or external command, operable program or batch file.
Here is how I solved my issue:
#echo off
set PARAM_HOTE=localhost
set PARAM_PORT=5435
set PSQL="C:\Program Files\PostgreSQL\9.2\bin\psql.exe"
set SQL_LISTE_VUES=^
select string_agg( concat('--table=' , quote_ident(nspname) , '.' , quote_ident(relname)), ' ' )^
from (^
select *^
from pg_class^
join pg_namespace on pg_namespace.oid = pg_class.relnamespace^
where relkind = 'v'^
and nspname = 'rec'^
order by relname ASC^
) infos_vues^
;
for /f "usebackq delims=" %%i in (`%%PSQL%% --quiet --tuples-only --host %PARAM_HOTE% --port %PARAM_PORT% --username "rec" -c "%SQL_LISTE_VUES%" REC`) do set LISTE_VUES=%%i
echo %LISTE_VUES%
I rewrote my query by replacing || with the concat function
I used back ticks
I escaped % with %% in the for command
Related
I'm trying to export my data to a csv file using the command below. I am using -s, for comma delimited.
It's working fine but I face one problem when my customer name has comma(,) inside. In this case it's cutting the customer name in two different column.
How do I resolve this issue?
sqlcmd -S . -d MYDB -E -Q "set nocount on; select 'customer_id','customer_name','salesrep_id'; select customer_id,customer_name,salesrep_id from customer where customer_id=106866" -b -o C:\customer.csv -h-1 -s, -w 700
It would be nice to know your datatypes. But generally put varchar in double quotes. Here's the very basic idea:
select customer_id, '"' + customer_name + '"', salesrep_id from customer where customer_id=106866
I generally like to explicitly create the csv line myself, so my selects usually look more like this:
select '"' + customer_id + '","' + customer_name + '"," + salesrep_id + '"' from customer where customer_id=106866
If there are columns that are numbers, then I convert the column to varchar and don't put double quotes around it.
Based on your comments:
Here is a .cmd file that I use in production to create a .csv file:
set proc=NoBillsReport
sqlcmd -E -S win11.net.davisbrownlaw.com -d ProLaw -Q "exec %proc%" -h -1 -W > "c:\2019\no-bills.csv"
It uses a stored procedure and redirects the output instead of trying to have SQL do the output.
Here is a simplified version of my stored procedure:
--Header row
select '"Matter ID","Client Sort","Matter Description","Billing Initials","Area of Law","Fees No-Billed","Costs No-Billed"'
-- Costs that were no-billed in previous month
select '"' + matterid + '","' + clientsort + '","' + shortdesc + '","'
+ initials + '","' + areaoflaw + '",' + convert(varchar(50),nobilled_fees)
+ ',' + convert(varchar(50),nobilled_costs)
from Matters
And its output:
"Matter ID","Client Sort","Matter Description","Billing Initials","Area of Law","Fees No-Billed","Costs No-Billed"
"70441233","ACME","Closing Wheel, Whenever","LA","IP",0.00,7.98
which works great as the contents of a .csv file.
The stored procedure is nice because it doesn't all have to fit on 1 long command line. But, for your use, I think this has a shot at working:
sqlcmd -S . -d MYDB -E -Q "set nocount on; select '""customer_id"",""customer_name"",""salesrep_id""'; select convert(varchar(50),customer_id) + ',""' + customer_name + '"",""' + salesrep_id + '""' from customer where customer_id=106866; " -h -1 -W > C:\customer.csv
Note that this is called from a .cmd file. The double double quotes might be different if called directly from the command line. Also, on my system, the 11 spaces after the ; at the end of the sql line is necessary due to how the double double quotes are consolidated. And yes, there are single quotes and double double quotes all over the place.
Based on further comments, here is how I script a .cmd file to pull year, month, day:
:: Get current year/month/day
for /f "skip=1 tokens=2-4 delims=(-)" %%a in ('"echo.|date"') do (
for /f "tokens=1-3 delims=/.- " %%A in ("%Date:* =%") do (
set %%a=%%A&set %%b=%%B&set %%c=%%C)
)
)
set /a "yy=10000%yy% %% 10000 %% 2000 + 2000,mm=100%mm% %% 100,dd=100%dd% %% 100"
:: Pad with leading zeros if needed
set mm=0%mm%
set mm=%mm:~-2%
set dd=0%dd%
set dd=%dd:~-2%
Then your filename would be "customer%yy%%mm%%dd%.csv"
I want to check if a certain environment variable is set in the PC. If yes do x if not do y.
I tried these and some variations of them:
IF EXISTS %SIGN% runtest.exe --redirect -l %NAME%
ELSE runtest.exe -l %NAME%
if "%SIGN%" == "" runtest.exe --redirect -l %NAME%
ELSE runtest.exe -l %NAME%
None of them work well in both cases (when the environment variable SIGN exists and when it doesn't exist).Sometimes just in one case...
Please can you help?
Thanks!
if exists checks for files.
For variables do: if defined sign (without the percent-signs)
IF Conditionally perform a command
IF DEFINED SIGN (
runtest.exe --redirect -l %NAME%
) ELSE (
runtest.exe -l %NAME%
)
or shortly
IF DEFINED SIGN (runtest.exe --redirect -l %NAME%) ELSE (runtest.exe -l %NAME%)
Valid syntax:
all ), ELSE and ( must be on an only line as follows: ) ELSE (
Note:
if DEFINED will return true if the variable contains any value (even if the value is just a space).
According to above predicate, IF DEFINED SIGN condition seems to be equivalent to reformulated test if NOT "%SIGN%"=="" but it is valid in batch only, where %undefined_variable% results to an empty string:
if NOT "%SIGN%"=="" (runtest.exe --redirect -l %NAME%) ELSE (runtest.exe -l %NAME%)
Otherwise, in pure CLI, %undefined_variable% results to %undefined_variable%
Proof:
==>type uv.bat
#echo undefined_variable="%undefined_variable%"
==>uv.bat
undefined_variable=""
==>echo undefined_variable="%undefined_variable%"
undefined_variable="%undefined_variable%"
==>
I encountered the following confusing problem:
A DB table was updated through CSV file and some KSH script:
#!/bin/ksh
moduleDir=$ACS_INSTALL/FMC_TER_AM_028/Database
logFile=$moduleDir/install_scripts/`basename $0`_$(date '+%Y%m%d_%H%M').log
(
cp -ri $moduleDir/ntscripts/datafiles/* $PROVHOME/database/ntscripts/datafiles
echo "\n***********************"
echo "* New/Modified files: *"
echo "***********************"
find $moduleDir/ntscripts/datafiles -type f | xargs ls -l
) 2>&1 | tee $logFile
echo "\nInstallation of Database completed\n"
The update is equal (under equal I mean that the visual result is the same as the INSERT command below) to the following INSERT query:
INSERT INTO tp_nt_mapping (TARIFF_PLAN, SERVICE_TYPE, NETWORK_TEMPLATE, IN_CREATE, IN_DELETE)
VALUES (171, 'Postpaid', '11', '', '')
When I'm using the following SELECT command:
SELECT * FROM tp_nt_mapping ORDER BY tariff_plan DESC
I'm able to see the new inserted record, but when I try with any of the following SELECT queries, I'm not:
SELECT * FROM tp_nt_mapping WHERE network_template = 11 ORDER BY tariff_plan DESC
SELECT * FROM tp_nt_mapping WHERE network_template = '11' ORDER BY tariff_plan DESC
Any suggestions?
The value in network_template field isn't just '11' but '11' || chr(13).
So you have the carrige return char at the end.
You can fix the data by doing:
update tp_nt_mapping
set network_template = replace(network_template, chr(13), '')
But better check why it was added on the first place....
Currently, i have a number of files stored in postgres 8.4 as bytea. The file types are .doc, .odt, .pdf, .txt and etc.
May i know how to download all the file stored in Postgres because i need to to do a backup.
I need them in their original file type instead of bytea format.
Thanks!
One simple option is to use COPY command with encode to hex format and then apply xxd shell command (with -p continuous hexdump style switch). For example let's say I have jpg image in bytea column in samples table:
\copy (SELECT encode(file, 'hex') FROM samples LIMIT 1) TO
'/home/grzegorz/Desktop/image.hex'
$ xxd -p -r image.hex > image.jpg
As I checked it works in practice.
Try this:
COPY (SELECT yourbyteacolumn FROM yourtable WHERE <add your clauses here> ...) TO 'youroutputfile' (FORMAT binary)
Here's the simplest thing I could come up with:
psql -qAt "select encode(file,'base64') from files limit 1" | base64 -d
The -qAt is important as it strips off any formatting of the output. These options are available inside the psql shell, too.
base64
psql -Aqt -c "SELECT encode(content, 'base64') FROM ..." | base64 -d > file
xxd
psql -Aqt -c "SELECT encode(content, 'hex') FROM ..." | xxd -p -r > file
If you have a lot of data to download then you can get the lines first and then iterate through each one writing the bytea field to file.
$resource = pg_connect('host=localhost port=5432 dbname=website user=super password=************');
// grab all the user IDs
$userResponse = pg_query('select distinct(r.id) from resource r
join connection c on r.id = c.resource_id_from
join resource rfile on c.resource_id_to = rfile.id and rfile.resource_type_id = 10
join file f on rfile.id = f.resource_id
join file_type ft on f.file_type_id = ft.id
where r.resource_type_id = 38');
// need to work through one by one to handle data
while($user = pg_fetch_array($userResponse)){
$user_id = $user['id'];
$query = 'select r.id, f.data, rfile.resource_type_id, ft.extension from resource r
join connection c on r.id = c.resource_id_from
join resource rfile on c.resource_id_to = rfile.id and rfile.resource_type_id = 10
join file f on rfile.id = f.resource_id
join file_type ft on f.file_type_id = ft.id
where r.resource_type_id = 38 and r.id = ' . $user_id;
$fileResponse = pg_query($query);
$fileData = pg_fetch_array($fileResponse);
$data = pg_unescape_bytea($fileData['data']);
$extension = $fileData['extension'];
$fileId = $fileData['id'];
$filename = $fileId . '.' . $extension;
$fileHandle = fopen($filename, 'w');
fwrite($fileHandle, $data);
fclose($fileHandle);
}
DO $$
DECLARE
l_lob_id OID;
r record; BEGIN
for r in
select data, filename from bytea_table
LOOP
l_lob_id:=lo_from_bytea(0,r.data);
PERFORM lo_export(l_lob_id,'/home/...'||r.filename);
PERFORM lo_unlink(l_lob_id);
END LOOP;
END; $$
Best I'm aware, bytea to file needs to be done at the app level.
(9.1 might change this with the filesystem data wrapper contrib. There's also a lo_export function, but it is not applicable here.)
If you want to do this from a local windows, and not from the server, you will have to run every statement individually, and have PGAdmin and certutil:
Have PGAdmin installed.
Open cmd from the runtime folder or cd "C:\Program Files\pgAdmin 4\v6\runtime"
Run in PGAdmin query to get every statement that you will have to paste in cmd:
SELECT 'set PGPASSWORD={PASSWORD} && psql -h {host} -U {user} -d {db name} -Aqt -c "SELECT encode({bytea_column}, ''base64'') FROM {table} WHERE id='||id||'" > %a% && CERTUTIL -decode %a% "C:\temp{name_of_the_folder}\FileName - '||{file_name}||' ('||TO_CHAR(current_timestamp(),'DD.MM.YYYY,HH24 MI SS')||').'||{file_extension}||'"'
FROM table WHERE ....;
Replace {...}
It will generate something like:
set PGPASSWORD=123 psql -h 192.1.1.1 -U postgres -d my_test_db -Aqt -c "SELECT encode(file_bytea, 'base64') FROM test_table_bytea WHERE id=33" > %a% && CERTUTIL -decode %a% "C:\temp\DB_FILE\FileName - test1 - (06.04.2022,15 42 26).docx"
set PGPASSWORD=123 psql -h 192.1.1.1 -U postgres -d my_test_db -Aqt -c "SELECT encode(file_bytea, 'base64') FROM test_table_bytea WHERE id=44" > %a% && CERTUTIL -decode %a% "C:\temp\DB_FILE\FileName - test2 - (06.04.2022,15 42 26).pdf"
Copy paste all the generated statements in CMD. The files will be saved to your local machine.
i am writing a short bat file that contours a xyz file with GMT utilities (generic mapping tool) i want to read the max and min file and use it later in the bat file what i did is
set max_color=gawk "BEGIN {max = 0} {if ($3>max) max=$3} END {print max}" %file%
set min_color=gawk "BEGIN {min = %max_color%} {if ($3'<'min) min=$3} END {print min}" %file%
but when i try reading it later
makecpt -Crainbow -T%min_color%/%max_color%/10 > conc.cpt
instead of reding the value it has the whole gawk one liner
how can i set a value
use a for loop to get the results of the gawk command, eg
for /f %%a in ('your gawk command') do (
set var=%%a
)