I am creating a string that I would like to pass to the bcp command from within a PowerShell script. Here is the code that creates the string
$fullFilePath = (pwd).path + "\$db.$table" + ".dat"
$copyExecString = "bcp $db.$table out $fullFilePath -E -N -Slocalhost"
I can paste the Write-host result of this into a command prompt and it runs just fine, but it won't work as follows:
$fullFilePath = (pwd).path + "\$db.$table" + ".dat"
$copyExecString = "$db.$table out $fullFilePath -E -N -Slocalhost"
bcp $copyExecString
I also tried using the ampersand character before calling the bcp command, but to no avail
Any ideas on how to achieve this functionality would be greatly appreciated
Instead of a single string, tokenize it by splitting the arguments out by spaces:
& bcp $db.$table out $fullFilePath -E -N -Slocalhost
you can use Invoke-Expression cmdlet to ensure that you parse what you want to execute prior to execution:
Invoke-Expression "bcp $copyExecString"
Related
This is a follow-up to my question here: How to write bash function to print and run command when the command has arguments with spaces or things to be expanded
Suppose I have this function to print and run a command stored in an array:
# Print and run the cmd stored in the passed-in array
print_and_run() {
echo "Running cmd: $*"
# run the command by calling all elements of the command array at once
"$#"
}
This works fine:
cmd_array=(ls -a /)
print_and_run "${cmd_array[#]}"
But this does NOT work:
cmd_array=(ls -a / | grep "home")
print_and_run "${cmd_array[#]}"
Error: syntax error near unexpected token `|':
eRCaGuy_hello_world/bash$ ./print_and_run.sh
./print_and_run.sh: line 55: syntax error near unexpected token `|'
./print_and_run.sh: line 55: `cmd_array=(ls -a / | grep "home")'
How can I get this concept to work with the pipe operator (|) in the command?
If you want to treat an array element containing only | as an instruction to generate a pipeline, you can do that. I don't recommend it -- it means you have security risk if you don't verify that variables into your string can't consist only of a single pipe character -- but it's possible.
Below, we create a random single-use "$pipe" sigil to make that attack harder. If you're unwilling to do that, change [[ $arg = "$pipe" ]] to [[ $arg = "|" ]].
# generate something random to make an attacker's job harder
pipe=$(uuidgen)
# use that randomly-generated sigil in place of | in our array
cmd_array=(
ls -a /
"$pipe" grep "home"
)
exec_array_pipe() {
local arg cmd_q
local -a cmd=( )
while (( $# )); do
arg=$1; shift
if [[ $arg = "$pipe" ]]; then
# log an eval-safe copy of what we're about to run
printf -v cmd_q '%q ' "${cmd[#]}"
echo "Starting pipeline component: $cmd_q" >&2
# Recurse into a new copy of ourselves as a child process
"${cmd[#]}" | exec_array_pipe "$#"
return
fi
cmd+=( "$arg" )
done
printf -v cmd_q '%q ' "${cmd[#]}"
echo "Starting pipeline component: $cmd_q" >&2
"${cmd[#]}"
}
exec_array_pipe "${cmd_array[#]}"
See this running in an online sandbox at https://ideone.com/IWOTfO
Do this instead. It works.
print_and_run() {
echo "Running cmd: $1"
eval "$1"
}
Example usage:
cmd='ls -a / | grep -C 9999 --color=always "home"'
print_and_run "$cmd"
Output:
Running cmd: ls -a / | grep -C 9999 --color=always "home"
(rest of output here, with the word "home" highlighted in red)
The general direction is that you don't. You do not store the whole command line to be printed later, and this is not the direction you should take.
The "bad" solution is to use eval.
The "good" solution is to store the literal '|' character inside the array (or some better representation of it) and parse the array, extract the pipe parts and execute them. This is presented by Charles in the other amazing answer. It is just rewriting the parser that already exists in the shell. It requires significant work, and expanding it will require significant work.
The end result is, is that you are reimplementing parts of shell inside shell. Basically writing a shell interpreter in shell. At this point, you can just consider taking Bash sources and implementing a new shopt -o print_the_command_before_executing option in the sources, which might just be simpler.
However, I believe the end goal is to give users a way to see what is being executed. I would propose to approach it like .gitlab-ci.yml does with script: statements. If you want to invent your own language with "debug" support, do just that instead of half-measures. Consider the following YAML file:
- ls -a / | grep "home"
- echo other commands
- for i in "stuff"; do
echo "$i";
done
- |
for i in "stuff"; do
echo "$i"
done
Then the following "runner":
import yaml
import shlex
import os
import sys
script = []
input = yaml.safe_load(open(sys.argv[1], "r"))
for line in input:
script += [
"echo + " + shlex.quote(line).replace("\n", "<newline>"), # some unicode like  would look nice
line,
]
os.execvp("bash", ["bash", "-c", "\n".join(script)])
Executing the runner results in:
+ ls -a / | grep "home"
home
+ echo other commands
other commands
+ for i in "stuff"; do echo "$i"; done
stuff
+ for i in "stuff"; do<newline> echo "$i"<newline>done<newline>
stuff
This offers greater flexibility and is rather simple, supports any shell construct with ease. You can try gitlab-ci/cd on their repository and read the docs.
The YAML format is only an example of the input format. Using special comments like # --- cut --- between parts and extracting each part with the parser will allow running shellcheck over the script. Instead of generating a script with echo statements, you could run Bash interactively, print the part to be executed and then "feed" the part to be executed to interactive Bash. This will alow to preserve $?.
Either way - with a "good" solution, you end up with a custom parser.
Instead of passing an array, you can pass the whole function and use the output of declare -f with some custom parsing:
print_and_run() {
echo "+ $(
declare -f "$1" |
# Remove `f() {` and `}`. Remove indentation.
sed '1d;2d;$d;s/^ *//' |
# Replace newlines with <newline>.
sed -z 's/\n*$//;s/\n/<newline>/'
)"
"$#"
}
cmd() { ls -a / | grep "home"; }
print_and_run cmd
Results in:
+ ls --color -F -a / | grep "home"
home/
It will allow for supporting any shell construct and still allow you to check it with shellcheck and doesn't require that much work.
I know how to setup a variable in a cmd file that passes the variable to SQL via sqlcmd.
Example:
sqlcmd -Usa -Ppass -d MASTER -v num="%num%" -i C:\scriptfile
My question is how can I define a variable in SQL that can be read outside of SQL. I know how to declare and define #variables in a SQL script but those are not recognized outside of when the SQL script runs.
My question is how to you pass a variable from SQL back to cmd?
Is there anyway to accomplish this?
Thank you
You can do this using scripting variable and using the -v option of SQLCMD utility. A small example from MSDN Documentation
Consider that the script file name is testscript.sql, Col1 is a scripting variable; your SQL script look like
USE test;
SELECT x.$(Col1) FROM Student x WHERE marks < 5;
You can then specify the name of the column that you want returned by using the -v option like
sqlcmd -v Col1 = "FirstName" -i c:\testscript.sql
Which will resemble to below query
SELECT x.FirstName FROM Student x WHERE marks < 5;
EDIT:
If you just want to capture the output from your script file then you can use -o parameter and specify a outfile like
sqlcmd -v Col1 = "FirstName" -i c:\testscript.sql -o output.txt
Thanks Rahul, you inadvertently answered my question. You can output your script results to a file via the -o option for SQLCMD.
Thinking about that I realized I could use the PRINT SQL to create a -o .cmd file that contains the .cmd syntax to define a variable. Then in the .cmd file I tell it to run the SQL created .cmd file and then the variable gets defined in the .cmd environment.
Kind of a round about way but works!!
Thanks!
If you just run a simple select to get your value or an exec spname that returns just the value you are after, you can use the following.
for /f "tokens=*" %a in ('sqlcmd -Usa -Ppass -W -h -1 -d MASTER -Q "select Column from table"') do set ResultVariable=%a
Remember to use %%a if putting this in a bat file
I'm having a hard time to write a simple batch file as powershell script.
Consider this folder structure. Note the directory with the cool [1] in it...
exiftool.exe
Is a command utility to (for example) extract pictures from embedded MP3 tags.
I uploaded its help if you need more info.
oldscript.cmd
exiftool -picture -b input.mp3 > output.jpg
This line is the one to write in powershell. I found the syntax in a forum post from the author
-picture stands for the tag to extract and -b stands for binary mode
input.mp3 is my test mp3 which can contain special characters in its path like [ and ]
> output.jpg defines the name and saves the resulting image in the same folder
newscript.ps1
My best current non-working code is:
$ownpath = Split-Path $MyInvocation.MyCommand.Path
$exe = $ownpath + '\exiftool.exe'
$input = $ownpath + '\input.mp3'
$outimg = $ownpath + '\output.jpg'
& $exe -picture -binary $input| Set-Content -literalPath $outimg -encoding UTF8
I found Set-Content which is able to handle special characters in pathes through "-literalpath". But I'm still not able to convert the batch to a Powershell script because
Set-Content (and Out-File method too) seems work different compared to old batch piping (">"). The resulting image is not viewable regardless which encoding I use. The help file from above says that exiftool is using UTF8 encoding.
Of course I tried other available encodings, but all of them failed to produce a viewable image. I'm stuck at this point. So my initial question still stands partly "How do I convert this batch file to powershell".
So why is it working when using the old batch command?
For example: create a folder "D:folder" and place this MP3 file with a cover image in it.
Download exiftool.exe from above and place it there too.
The old batch command will work and give you a viewable image
D:\folder\exiftool -picture -binary D:\folder\input.mp3 > D:\folder\output.jpg
The new Powershell V2 script with the same syntax will fail. Why?
& D:\folder\exiftool.exe -picture -binary D:\folder\input.mp3 > D:\folder\output.jpg
You can try this, though I've not tested it 'cause I've not an mp3 with embedded images:
$file = & "D:\folder\exiftool.exe" -picture -binary "D:\folder\input.mp3"
[io.file]::WriteAllBytes('D:\folder\input[1].jpg',$file)
Edit:
using this line from a powershell console return a readable image:
cmd.exe /c "D:\folder\exiftool.exe -picture -binary `"D:\folder\input.mp3`" > image.jpg"
You can use special characters in path and in file name as:
$exe = "c:\ps\exiftool.exe"
$mp3 = "c:\ps\a[1]\input.mp3"
$jpg = " c:\ps\a[1]\image[1].jpg"
cmd.exe /c "$exe -picture -binary $mp3 > $jpg"
with spaces inside path:
$exe = "c:\ps\exiftool.exe"
$mp3 = "`"c:\ps\a [1]\input.mp3`""
$jpg = "`"c:\ps\a [1]\image [1].jpg`""
cmd.exe /c "$exe -picture -binary $mp3 > $jpg"
Try this:
& $exe -picture -b $input | Out-File -LiteralPath $output
There is no need to complicate things by using Start-Process. Because you compute the path to the exe and put that result in a string, you only need use the call operator & to invoke the command named by the string that follows it.
Here is a work around. It seems you can't avoid good old cmd.exe completely.
Thanks should go # C.B.
$ownpath = Split-Path $MyInvocation.MyCommand.Path
$exe = $ownpath + '\exiftool.exe'
$input = $ownpath + '\input.mp3'
$output = $ownpath + '\output.jpg'
cmd.exe /c " `"$exe`" -picture -binary `"$input`" > `"$output`" "
Note:
This way all pathes can contain special characters like [ and ] or spaces
That extra space in " `"$exe is important. It won't work without it.
The normal Powershell way with set-content, Out-File (">" is an alias) and [io.file]::WriteAllBytes all don't work with the exiftool.exe utility. For me its a miracle.
I'm running the Microsoft SQLCMD tool for Linux (CTP 11.0.1720.0) on a Linux box (Red Hat Enterprise Server 5.3 tikanga) with Korn shell. The tool is properly configured, and works in all cases except when using scripting variables.
I have an SQL script, that looks like this.
SELECT COLUMN1 FROM TABLE WHERE COLUMN2 = '$(param1)';
And I'm running the sqlcmd command like this.
sqlcmd -S server -d database -U user -P pass -i input.sql -v param1="DUMMYVALUE"
When I execute the above command, I get the following error.
Sqlcmd: 'param1=DUMMYVALUE': Invalid argument. Enter '-?' for help.
Help lists the below syntax.
[-v var = "value"...]
Am I missing something here?
You don't need to pass variables to sqlcmd. It auto picks from your shell variables:
e.g.
export param1=DUMMYVALUE
sqlcmd -S $host -U $user -P $pwd -d $db -i input.sql
In the RTP version (11.0.1790.0), the -v switch does not appear in the list of parameters when executing sqlcmd -?. Apparently this option isn't supported under the Linux version of the tool.
As far as I can tell, importing parameter values from environment variables doesn't work either.
If you need a workaround, one way would be to concatenate one or more :setvar statements with the text file containing the commands you want to run into a new file, then execute the new file. Based on your example:
echo :setvar param1 DUMMYVALUE > param_input.sql
cat input.sql >> param_input.sql
sqlcmd -S server -d database -U user -P pass -i param_input.sql
You can export the variable in linux. After that you won't need to pass the variable in sqlcmd. However, I did notice you will need to change your sql script and remove the :setvar command if it doesn't have a default value.
export dbName=xyz
sqlcmd -Uusername -Sservername -Ppassword -i script.sql
:setvar dbName --remove this line
USE [$(dbName)]
GO
I think you're just not quoting the input variables correctly. I created this bash script...
#!/bin/bash
# Create a sql file with a parameterized test script
echo "
set nocount on
select k = '-db', v = '\$(db)' union all
select k = '-schema', v = '\$(schema)' union all
select '-', 'static'
go" > ./test.sql
# capture input variables
DB=$1
SCHEMA="${2:-dbo}"
# Exec sqlcmd
sqlcmd -S 'localhost\lemur' -E -i ./test.sql -v "db=${DB}" -v "schema=${SCHEMA}"
... and tested it like so:
$ ./test.sh master
k v
------- ------
-db master
-schema dbo
- static
SQLCMD supports the -s parameter to specify the column separator, but I couldn't figure how how to represent the tab (CHAR(9)) character. I have tried the following but both don't work:
sqlcmd -S ServerName -E -Q"select * from mytable" -s"\t" -o results.txt
sqlcmd -S ServerName -E -Q"select * from mytable" -s'\t' -o results.txt
Any ideas how to do this in SQLCMD?
In a batch file, putting a tab between the double quotes works.
sqlcmd -S ServerName -E -Q"select * from mytable" -s" " -o results.txt
to do the same in a PowerShell file use escaped double quotes wrapped around an escaped tab
sqlcmd -S ServerName -E -Q"select * from mytable" -s `"`t`" -o results.txt
It's difficult to get unformatted results from SQLCMD.
If you want to create a tab-delimited output file, BCP might be a better bet:
bcp "select * from mytable" queryout results.txt -S server -T -c
Found a good answer here: SQLCMD outfile as tab delimited text file
Open Notepad
Paste this: sqlcmd -S (local) -E -s"<TAB>" -Q "select * from sys.dm_exec_query_stats" -o MyOutput.txt -h-1 -W
Highlight <TAB>, then hit the Tab key
Save the file as MyBatch.bat
Run MyBatch.bat
I've tried numerous times to pass the actual TAB character in to SQLCMD, and I simply can't get it to take it. My favorite work-around to-date is to pass SQLCMD the ASCII "Unit Separator", which is hex 0x1F, and can be entered on the command line by typing Ctrl-_ (control underscore, which on a US keyboard becomes ctrl-shift-'-' (the '-' next to the '0' on the top row of the keyboard).
The advantage of using the 'Unit Separator' is that is is HIGHLY unlikely to be present in text of any description, and was designed specifically for this purpose (see https://en.wikipedia.org/wiki/Delimiter)
Having got SQLCMD to do that for me, I then pipe it's output though a Unix-style translate command as:
tr '\037' '\t'
The \037 is octal for the 'Unit Separator', and \t represents the tab character, 'tr' will translate BOTH of these for us, we don't need to rely on any quoting tricks in our scripts or shells.
To get 'tr' on windows, you can install the CoreUtils package from GnuWin32 (see http://gnuwin32.sourceforge.net/packages/coreutils.htm) or go heavy-weight and install a full Unix environment such as Cygwin (http://cygwin.com/).
Putting the two together we get:
sqlcmd ... -h-1 -W -k -r1 -s^_ ... | tr '\037' '\t'
and this will give you your output with tabs.
Look up the other options I've used above, they're essential for trying to get clean output from SQLCMD (in order; no headers, trim white-space, CRLF to spaces, errors to STDERR (not your output file!) and the '^_' is how the Unit Separator will appear on the command line). You'll also need to add "SET NOCOUNT ON;" to your query or sql script, otherwise you'll get the row-count as a trialling message appearing in your output!
A similar answer to one posted above, but it's simpler in a way that I think is significant.
Open your text editor
Press Tab
Highlight the chunk of whitespace (the tab) created
Copy and paste that into the spot in your SQL command
Even though this tab is represented as a wide chunk of whitespace, it is a single character.
The other answer had some unnecessary stuff about pasting the whole command with "<TAB>" in it. I think that throws people off (it certainly threw me off).
To work in the Command Prompt window instead in batch file, this is the only way that I have found to solve it:
sqlcmd -S ServerName -E -d database_Name -Q"select col1, char(9), col2, char(9), col3, char(9), col4, char(9), col5 from mytable" -o results.txt -W -w 1024 -s "" -m 1
tldr: use ALT+009 the ascii tab code for the separator character
In the example, replace {ALTCHAR} with ALT+009 (hold the ALT key and enter the digits 009)
sqlcmd -E -d tempdb -W -s "{ALTCHAR}" -o junk.txt -Q "select 1 c1,2 c2,3 c3"
Edit junk.txt. Tabs will be between columns.
For other command line options:
sqlcmd -?
Note: The shell converts the ALT char to ^I, but if you try the command by typing -s "^I", you won't get the same results.
Use dynamic sql with CHAR(9):
SET #cmd ='SQLCMD -S MyServer -d MyDatabase -E -W -Q "SELECT * FROM MyTable" -s"' + CHAR(9) + '" -o "MyFilePath.txt"'
Try using horizontal scroll bars with cmd.exe or powershell. Right click shortcut and click properties for repeated use, or right click title bar and click properties after opening then click layout tab. In screen buffer size set width and height to 8000 and then unselect wrap text output on resize (important). Click ok. Then restore down by clicking button next to minimize. You should see horizontal and vertical scroll bars. You can maximize window now and scroll in any direction. Now you can see all records in database.
I had this problem while trying to run sqlcmd on terminal. I got it working by entering a tab character (copying from text editor didn't work for me).
Press cntrl + v then tab.
How to enter a tab char on command line?
To achieve this using sqlcmd you need to use the Tab character like so: \t
An example query exporting a single sql database table into a text file using a tab delimiter is as follows:
sqlcmd -S ServerName -d databaseTableName -Q "SELECT * FROM TABLE_NAME" -o C:\backups\tab_delimiter_bakup.txt -s"\t"
If on Linux then this will work as the -s (col_separator)
-s "$(printf "t" | tr 't' '\t')" or
SQLCMDCOLSEP="$(printf "t" | tr 't' '\t')" sqlcmd -S ...