I'm running a powershell script to get the results of a SQL query (in JSON format from SQL2016) and the results come back broken up into individual lines with '...' on the end instead of one JSON string and some header info at the top of the file. This makes the JSON unuseable.
I verified that this is on the PowerShell side by running the same query in SSMS and the results came out as expected (valid JSON)
I couldn't find any command line arguments for controlling the output of Invoke-SqlCommand
I'm new to PowerShell... Any ideas how to help me get clean JSON from this PowerShell script?
The powershell script:
Invoke-Sqlcmd -InputFile "C:\Dashboard\sql\gtldata.sql" | Out-File -filepath "C:\Dashboard\json\gtldata.json"
A sample of the returned document:
JSON_F52E2B61-18A1-11d1-B105-00805F49916B
-----------------------------------------
{"KPI":[{"BusinessUnit":"Water - Industrial","Location":"SPFIN","TestDate":"2016-09-19T21:11:10.837","TestResult":"Fail","FailReason":"P...
ial":"100161431","PumpType":"xxx","Stages":0},{"BusinessUnit":"Water - Industrial","Location":"SPFIN","TestDate":"2016-09-20T01:48...
"PumpType":"xxx","Stages":0},{"BusinessUnit":"Pre-engineered","Location":"SPSPA","TestDate":"2016-09-20T10:46:38.403","TestResult"...
You need to add the -MaxCharLength option to your invocation. By default, sqlcmd (and thus Invoke-Sqlcmd) use an 80-character output width, as cited in this sqlcmd documentation:
-w column_width
Specifies the screen width for output. This option sets the sqlcmd scripting variable SQLCMDCOLWIDTH. The column width must be a number greater than 8 and less than 65536. If the specified column width does not fall into that range, sqlcmd generates and error message. The default width is 80 characters. When an output line exceeds the specified column width, it wraps on to the next line.
I'd try setting -MaxCharLength 65535 for your output, since unformatted JSON will not have any line breaks in it.
Try the below as It worked for me. The result was a non truncated output of a varbinary column to string
original post [here][1]
bcp "SELECT CAST(BINARYCOL AS VARCHAR(MAX)) FROM OLTP_TABLE WHERE ID=123123 AND COMPANYID=123" queryout "C:\Users\USER\Documents\ps_scripts\res.txt" -c -S myserver.db.com -U admin -P password
[1]: https://stackoverflow.com/questions/60525910/powershell-truncating-sql-query-output?noredirect=1#comment107077512_60525910
Related
I am using below command to write data to csv file in isql
$ISQL -S DSA1_PROD -U emer_r_gh5432 -X
Query -
Select * from SecDb..LoginOwnerTb where SvrId= 45566 and OwnerRitsId = '1001167635';
OUTPUT TO '/tmp/sometable.csv' FORMAT ASCII DELIMITED BY ';' QUOTE '';
go
it says
Server 'ABC', Line 1:
Incorrect syntax near ';'.
Please help
NOTE: I'm assuming you're working with Sybase ASE and the isql command line tool. There may be other ways to accomplish what you're trying to do when going against the SQLAnywhere, IQ and/or Advantage database products ... *shrug* ...
The OUTPUT TO clause is used with the dbisql GUI tool.
To perform a somewhat-similar OUTPUT operation with the isql command line tool:
-- once logged in via isql ...
-- to write to new file; to overwrite existing file:
select ....
go > /path/to/local/file/accessible/by/user/running/isql
-- to append to existing file:
select ...
go >> /path/to/local/file/accessible/by/user/running/isql
To set the column delimiter you can use the -s flag when invoking isql from the command line, eg:
# set the column delimiter to a semi-colon:
$ isql ... -s ';' ...
# set the column delimiter to a pipe:
$ isql ... -s '|' ...
Keep in mind that the output will still be generated using fixed-width columns, with each column's width determined by either a) the column's datatype 'width' or b) the column's title/label width, whichever is wider.
I'm not aware of any way to perform the following with the isql command line tool:
designate a column delimiter on-the-fly while inside a isql session
designate a quote character
remove extra spaces (ie, ouput data in true delimited format as opposed to fixed-width format)
To generate true delimited files you have a few options:
see if the dbisql GUI tool serves your purpose [I don't use dbisql so I'm *assuming* the OUTPUT TO clause works as expected]
use the bcp (command line) utility to place the data into a delimited file [bcp options, and how to handle subsets of tables, is a much larger discussion, ie, too much to address in this response]
see if you can find another (3rd party) tool that can extract the desired data set to a delimited file
I was wondering if anyone can help.
I have a number of queries in SQL (all in separate *.sql files). I wanted to know if there is a way to run these queries automatically or mass run them to be saved to either a csv or txt file?
Also, I have come variables within these queries which will need to be amended on a weekly bases before the queries are run.
Thanks.
KJ
Could you please provide some additional help in relation to the variables? Previously I would declare and set variables as:
DECLARE #TW_FROM DATETIME
DECLARE #TW_TO DATETIME
SET #TW_FROM = '2015-11-16 00:00:00';
SET #TW_TO = '2015-11-22 23:00:00';
How do I do this using sqlcmd?
Yes, you can use sqlcmd to do this.
First of all - variables. You can refer to your variables in the .sql files using $(variablename) wherever you want to substitue the variable. For example,
use $(dbname);
select $(columnname) from table1 where column= '$(var1)'
You then call sqlcmd with the following command (note the argument -v variables)
sqlcmd -S servername -d database -i "yoursqlfile.sql" -v dbname="database" columnname="column" var1="Fred"
In order to output this to a file, you tag > filename.txt on the end
sqlcmd -S servername -d database -i "yoursqlfile.sql" -v dbname="database" columnname="column" var1="Fred" > filename.txt
If you want to output to a csv, you can also specify the delimiter using the argument -s (note the idfference with the capital S for server). So now we have
sqlcmd -S servername -d database -s "," -i "yoursqlfile.sql" -v dbname="database" columnname="column" var1="Fred" > filename.csv
If you want to output several commands to the same csv or txt file, use >> instead of > as it add to teh bottom of the file, rather than replacing it.
sqlcmd -S servername -d database -s "," -i "yoursqlfile.sql" -v dbname="database" columnname="column" var1="Fred" >> filename.csv
To run this for several scripts, you can put the statements in a batch file, and then change the variables every week.
You could write a batch file that uses sqlcmd:
MSDN sqlcmd
That will allow you to call script files in a loop and output the results to a file.
Convert your current scrips to a Stored Procedure.
You can then pass your variables to that and run the query.
If you have SQL Server agent available (SQL standard or better) you can use this to automate the running of the stored procedures.
Otherwise the same can be achieved with Task Scheduler in windows.
As for exporting to CSV this will be useful.
It depends on where your SQL Server is acutally running. It might be quite tricky to write anything to the location you want.
You could read about BCP.
My suggestion is:
Create an UDF (best is inline-UDF!) from all of your queries within your database. Than call them from EXCEL or any other fitting product. You might want to set up an Excel where all your queries are filled one on each Sheet automatically
i have looked all over the internet and cant seem to find a solution to this problem.
i am trying to output query results as a CSV through using a combination of sqlcmd and windows batch. here is what i have so far:
sqlcmd.exe -S %DBSERVER% -U %DBUSER% -P %DBPASS% -d %USERPREFIX% -Q "SELECT Username, UserDOB, UserGender FROM TABLE" -o %USERDATA%\%USERPREFIX%\FACT_BP.CSV -h-1 -s","
is there something i'm missing here? some setting that only looks at the first column of the query results?
any advice at all would be a huge help - i'm lost.
Here is the reference page from MSDN on SQLCMD.
http://technet.microsoft.com/en-us/library/ms162773.aspx
I placed this command in a batch file in C:\temp as go.bat.
sqlcmd -S(local) -E -dmaster
-Q"select cast(name as varchar(16)), str(database_id,1,0), create_date from sys.databases"
-oc:\temp\sys.databases.csv -h-1 -s,
Notice I hard coded the file name and removed the "" around the field delimiter.
I get the expected output below.
Either the command does not like the system variables or something else is wrong. Please try my code as a base line test. It works for SQL 2012.
Also, the number of lines is always dumped to file. You must clear this out of the file. That is why I do not use SQLCMD for ETL.
Why not use BCP instead?
I have writing several articles on my website.
http://craftydba.com/?p=1584
The Environment:
Solaris 10, /bin/sh script using sqsh and freetds to talk to an MS SQL Server
The (TLDR) Problem:
I need sqsh to ignore a dollar sign and pass it through to MS SQL.
The Background:
I'm writing some code that dynamically builds SQL to alter existing indexes but when it runs, I get errors:
Msg 2727, Level 11, State 1
Server 'HOST\SERVER', Line 1
Cannot find index 'foo'.
I dig around and see that there is no index named foo but there is one named foo$bar.
The built-up input SQL input looks fine...
% grep 'foo' input.sql
alter index [foo$bar] on ...
...and running this SQL through a New Query session or a third party app succeeds. However, the results when passed through sqsh don't see past that dollar sign:
% sqsh -e -i input.sql -o output.sql
% grep 'foo' output.sql
1> alter index [foo] on ...
...which suggests it's interpreting $bar as a variable or something.
Anyone know how to get sqsh to escape a dollar sign or allow one to pass through? I've tried various combinations of quotes and backslashes.
Help me, stackoverlow. You're my only hope.
Another option is to disable buffer expansion altogether by executing:
\set expand=0
on the sqsh prompt, or specify this command in the .sqshrc file, or start sqsh with the parameter
sqsh -e -i input.sql -o output.sql -Lexpand=0
If expansion is on (default) then sqsh will substitute the variable $bar with its contents.
Reference manual on page 5 states: "Note that in order to prevent the expansion of a variable use either single quotes, or two \’s, like thus:
1> \echo \\$name
$name
So, I believe that to prevent sqsh to substitute $bar with an empty string, you have to write something like:
alter index [foo\\$bar] on ...
alter index ['foo$bar'] on ...
To test it you can try first with SELECT 'foo\\$bar' = 1 or something similar.
Is there a way to send a osql query from the command line that returns results that do not include the big long line of dash characters that represent the column width?
Here is the sample code I am working with:
echo.
"%PROGRAMFILES%\Microsoft SQL Server\%SQLVER%\Tools\BINN\osql" -S . -E -Q "SELECT name + ', ' FROM sysdatabases order by name for XML PATH('')"
And the results look something like this:
-------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
master, model, msdb, openfire, tempdb,
Use -h-1 switch to suppress headers: the dashes are the header/data separator "line"
The same switch can be used for sqlcmd too