Is it possible to have a persistent psql connection in bash? - database

I have a very large bash script and I make thousands of psql queries throughout it which I suspect is slowing it down. I have created this MWE which proves my theory. I am assuming this slow down is because of having to connect to the db repeatedly. Is there a way to remain connected to psql in bash?
#!/bin/bash
DB_NAME=testdb
DB_USER=user1 #UPDATE THIS
#DROP DB
SQL_QUERY="DROP DATABASE IF EXISTS $DB_NAME;"
echo $SQL_QUERY | sudo -u $DB_USER psql >/dev/null
#CREATE DB
SQL_QUERY="CREATE DATABASE $DB_NAME;"
echo $SQL_QUERY | sudo -u $DB_USER psql >/dev/null
#CREATE TABLE
SQL_QUERY="CREATE TABLE foo
(
id BIGSERIAL PRIMARY KEY,
problems INTEGER
);"
echo $SQL_QUERY | sudo -u $DB_USER psql -d $DB_NAME >/dev/null
TARGET_ITERATIONS=1000
#MULTIPLE DB CALLS
SQL_QUERY="INSERT INTO foo
(
problems
)
VALUES
(
99
);"
START_TIME=$(date +%s)
for ITERATION in $(seq $TARGET_ITERATIONS)
do
echo $SQL_QUERY | sudo -u $DB_USER psql -d $DB_NAME >/dev/null
done
STOP_TIME=$(date +%s)
echo "Multiple Call Duration (s): $((STOP_TIME-START_TIME))"
#Single db call
SQL_QUERY=""
for ITERATION in $(seq $TARGET_ITERATIONS)
do
SQL_QUERY+="INSERT INTO foo
(
problems
)
VALUES
(
99
);"
done
START_TIME=$(date +%s)
echo $SQL_QUERY | sudo -u postgres psql -d $DB_NAME >/dev/null
STOP_TIME=$(date +%s)
echo "Single Call Duration (s): $((STOP_TIME-START_TIME))"
Output:
$ ./test_psql.sh
Multiple Call Duration (s): 64
Single Call Duration (s): 12

Per your comment, try running this:
#/bin/bash
mkfifo /tmp/mypipe
chmod a+r /tmp/mypipe
sleep 10000 > /tmp/mypipe &
psql -f /tmp/mypipe &
echo "\pset pager off" > /tmp/mypipe
echo "select count(*) from information_schema.columns;" > /tmp/mypipe
echo "select count(*) from information_schema.columns;" > /tmp/mypipe
sleep 3
echo "select count(*) from information_schema.columns;" > /tmp/mypipe
echo "\q" > /tmp/mypipe
rm /tmp/mypipe

Related

bash script stuck in single line

I have below command for bash script:
#!/bin/sh
sqlserver=("localhost" "testserver")
db="master"
user="sa"
pass="test#12345"
for item in "${sqlserver[#]}"
do
sqlcmd -S "$item" -d $db -U $user -P $pass -I -h-1 -Q "set nocount on;SELECT name as username FROM SYSUSERS WHERE NAME LIKE '%test%'" >> "$item".txt
while IFS= read -r line; do
if [ -n "$line" ]
then
#echo "Text read from file: $line"
sqlcmd -S "$item" -d $db -U $user -P $pass -I -b -h-1 -Q "set nocount on;drop user $line"
fi
done < "$item".txt
done
script is stuck at sqlcmd -S $item -d $DATABASE -U $USERNAME -P $PASSWORD -I -b -h-1 -Q "set nocount on;drop user $line". and this is going to in loop , it's goes infinity. how to get out from this if command.
I got the error as below:
Msg 15151, Level 16, State 1, Server 66752ccaf826, Line 1
Cannot drop the user 'test2', because it does not exist or you do not have permission.
Msg 15151, Level 16, State 1, Server 66752ccaf826, Line 1
Cannot drop the user 'test2', because it does not exist or you do not have permission.
Msg 15151, Level 16, State 1, Server 66752ccaf826, Line 1
Cannot drop the user 'test2', because it does not exist or you do not have permission.
Msg 15151, Level 16, State 1, Server 66752ccaf826, Line 1
Cannot drop the user 'test2', because it does not exist or you do not have permission.
Issue: How to break or exit loop once end of file? and after complete file it will go with 2nd sql server with continue for loop.
For create User:
CREATE USER test1 WITHOUT LOGIN;
CREATE USER test2 WITHOUT LOGIN;
Update1:
If I use echo "Text read from file: $line" then if condition running properly but when I use sqlcmd -S "$item" -d $db -U $user -P $pass -I -b -h-1 -Q "set nocount on;drop user $line" then it goes to infinite loop.
Assuming you want the while loop to end with the end-of-file of "$item.txt", then you need to change
if [[ ! -z $line ]]
to (for bash)
if [[ -z $line ]] ; then break ; fi
or (for sh)
if [ -z $line ] ; then break ; fi
and remove the other "fi".
As for the command
sqlcmd -S $item -d $DATABASE -U $USERNAME -P $PASSWORD -I -b -h-1 -Q "set nocount on;drop user $line"
you need to resolve the issue reported, from the shell command line, before attempting to do that within the script.

Can PSCP retrieve file from multiple servers using an array of ip and looping?

I get more than one remote source not supported error on pscp if my script is written like this (no issues with plink) :
I want to retrieve files from multiple UNIX servers to local windows
Could someone help me to verify my code?
#Server Information:
$Server_IP=#("root#192.168.13.10","root#192.168.13.11")
$PPK_Path="C:\Users\me\Desktop\private-key.ppk"
#Local machine related information
$Dest_Path=#("C:\Users\me\Desktop\savehere01\","C:\Users\me\Desktop\savehere02\")
#Commands //Change with cautious
For ($i=0; $i -le 2; $i++) {
#Prompt computer to start plink.exe to insert private key and enable ssh
Echo "n" | plink -ssh -i $PPK_Path $Server_IP[$i]
#Prompt Powershell to run scp
pscp -r $Server_IP[$i]:/cf/conf/backup/* $Dest_Path[$i]
}
However, if i run my script as below, i am able to retrieve files from multiple servers to one single local host.
Echo "y" | plink -ssh -i C:\Users\me\Desktop\private-key.ppk root#192.168.13.32
pscp -pw testing -r root#192.168.13.10:/cf/conf/backup/* C:\Users\me\Desktop\savehere\
Echo "y" | plink -ssh -i C:\Users\me\Desktop\private-key.ppk root#192.168.13.11
pscp -pw testing -r root#192.168.13.11:/cf/conf/backup/* C:\Users\me\Desktop\savehere02\
EDIT
foreach ($IP in $Server_IP){
#Prompt computer to start plink.exe to insert private key and enable ssh
Echo "y" | plink -ssh -i $PPK_Path $IP
#Prompt Powershell to run pscp
pscp -pw testing -r $IP":"/cf/conf/backup/* C:\Users\me\Desktop\savehere\
}
I found some erros in your code:
1)Put this line
$Server_Username[$i]+"#"+$Server_IP[$i]
instead
${Server_Username[$i]}#${Server_IP[$i]}
2)You have only 2 elements in array
your loop must be
($i=0; $i -lt 2; $i++)
3)You can use just this construction
$Dest_Path[$i]
instead
${Dest_Path[$i]}
Additional:
#Commands //Change with cautious
For ($i=0; $i -lt 2; $i++) {
#Prompt computer to start plink.exe to insert private key and enable ssh
Echo "n" | plink -ssh -i $PPK_Path $Server_IP[$i]
#Prompt Powershell to run scp
pscp -r $Server_IP[$i]:/cf/conf/backup/* $Dest_Path[$i]
}
Try this fix

How can I get this script to loop X amount of times?

I am trying to loop everything under groupadd coffee for 5 times for creating 5 new users at a time. Taking those variables and filling in the other commands. After all that is finished start again for a new user. I've searched up and down and can't seem to figure it out. Here is the script:
#!/bin/bash
#Script for adding users and groups
if [ $(getent group Coffee) ]; then
echo
else
sudo groupadd Coffee
fi
read -p "What is the new user's login ID? " username
read -s -p "What is the user's password? " password
echo
read -p "What is the new user's full name? " fullname
read -p "What is the new user's initial group? " intgroup
read -p "What is the new user's additional group? " addgroup
if [ $(getent group $intgroup) ]; then
echo
else
sudo groupadd $intgroup
fi
if [ $(getent group $addgroup) ]; then
echo
else
sudo groupadd $addgroup
fi
sudo useradd -m -G Coffee,$intgroup,$addgroup $username
if [ -z $password ]
then
echo -e "newpass\nnewpass" | sudo passwd $username
else
echo -e "$password\n$password" | sudo passwd $username
fi
I know I may have to restructure this but I am hoping it is something simple I can add to fix it.
Here is the solution that I got helped with.
<pre>
#!/bin/bash
sudo groupadd Coffee
# adds group Coffee to system
COUNTER=0
while [ $COUNTER -lt 5 ]; do
echo The counter is $COUNTER
echo
read -p "What is the new user's login ID? " username
read -s -p "What is the user's password? " password
echo
read -p "What is the new user's full name? " fullname
read -p "What is the new user's initial group? " intgroup
read -p "What is the new user's additional group? " addgroup
#prompts for variables
if [ $(getent group $intgroup) ]; then
echo
else
sudo groupadd $intgroup
fi
#checks to see if specified group is on the system
if [ $(getent group $addgroup) ]; then
echo
else
sudo groupadd $addgroup
fi
#checks to see if specified group is on the system
sudo useradd -m -G Coffee,$intgroup,$addgroup $username
#creates home directory and adds user into groups
if [ -z $password ]
then
echo -e "newpass\nnewpass" | sudo passwd $username
else
echo -e "$password\n$password" | sudo passwd $username
fi
#changes password or gives default one
let COUNTER=COUNTER+1
done
echo
<code>

BATCH FILE : not able to display variable value by accessing by %variable%

I am using widows XP OS and have batch file where I set few variable values.
when i echo those varibales i can see the values, but when i use it in some commands i get empty sting as its value.
Sample Batch file
#ECHO OFF
SET "output=select * from employee where empid='160'"
CALL SET output=%%output:'=''%%
ECHO "%output%"
sqlcmd -b -h-1 -m-1 -V1 -S testsvr -E -Q "%output%' " -d tesdb
o/p select * from employee where empid=''160''
and value of variable in sqlcmd command is empty space.
I'm not sure, what you want to do.
#ECHO OFF
SET output=select * from employee where empid="160"
ECHO %output%
ECHO sqlcmd -b -h-1 -m-1 -V1 -S testsvr -E -Q '%output%' -d tesdb
..output is:
select * from employee where empid="160"
sqlcmd -b -h-1 -m-1 -V1 -S testsvr -E -Q 'select * from employee where empid="160"' -d tesdb
Try to place the quotes as follows:
SET output="select * from employee where empid='160'"
sqlcmd -b -h-1 -m-1 -V1 -S testsvr -E -Q %output% -d tesdb

How can I skip one file in a filesets?

Below is my code in batchfile:
for %%f in (%~dp0*.sql) do (
sqlcmd -S %SName% -U %UName% -P %Pwd% -d %DbName% -I -i "%%f" >>TsDeploy.txt 2>&1
)
the question is there is one file in that set must be NOT Run as first one.(cause the others is script file about create Table,that one is to insert data into table ).
How Should I do to achieve the goal in ONLY ONE batchfile?
You could rename the sql file that must not run first so that it sorts to the end (prefix with z?).
Or you could do something like
for %%f in (%~dp0*.sql) do (
if "%~nxf" neq "fileNameNotToRunFirst.sql" (
sqlcmd -S %SName% -U %UName% -P %Pwd% -d %DbName% -I -i "%%f" >>TsDeploy.txt 2>&1
)
)
sqlcmd -S %SName% -U %UName% -P %Pwd% -d %DbName% -I -i "%~dp0fileNameNotToRunFirst.sql" >>TsDeploy.txt 2>&1
Or you could create a master sql script that calls each of the others in the proper order, as described in How to Run a Series of T-SQL Scripts in a Specific Order

Resources