Powershell: Text file update increase big file size - sql-server

I wirte a XML file using SQL Server query result and I need to input a first line to ensure necessary head of file to integration of customer system.
I have sucess to create and update it, but the file is very large, for example from 2.4 MB to 120 MB.
Its caused by many blank lines the powershell insert when file is created.
See the script below.
function ProcToXmlFile {
param
(
[string]$outputType,
[string]$filename
)
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=localhost;Database=customerdatabase;Integrated Security=True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$Query = "select * from table where condition=1"
FOR
XML PATH('root'),
TYPE
)
FOR XML PATH(''),
ROOT('root')"
$SqlCmd.CommandText = $Query
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$SqlConnection.Close()
#$DataSet.Tables[0] | Format-Wide -AutoSize -Force
#$DataSet = -replace "\s", ""
$DataSet.Tables[0] | Format-Wide -AutoSize
$Headerstring = '<?xml version="1.0"?>'
$Headerstring | Out-File $filename
if ($outputType -eq "Text")
{
$DataSet.Tables[0] | Format-Table -HideTableHeaders -autosize -Wrap | out-file $filename -Encoding utf8
}
Output file (the first line is the problem because I need to make this script):
<?xml version="1.0"?>
<publicacoes>
<publicacao>
<codigo>1111111</codigo>
<rec_id>2022/06/28 08:26</rec_id>
<data_publicacao>2022/06/28 00:00</data_publicacao>
<pacote>Lorem ipsum dolor sit amet</pacote>
<caderno>Lorem ipsum dolor sit amet</caderno>
<pesquisa>Lorem ipsum dolor sit amet</pesquisa>
<detalhamento>Lorem ipsum dolor sit amet, consectetur adipiscing elit.</detalhamento>
<processo>11111111111111</processo>
<pagina>null</pagina>
<texto>Lorem ipsum dolor sit amet, consectetur adipiscing elit.</texto>
</publicacao>
</publicacoes>

So, you've clearly dummied out your SQL query partially. That's fine, but it means I'm guessing here and you'll have to account for that.
I am assuming your output has a single column of output, since that's typically how FOR XML works. The core problem here is likely that SQL Server assigns a random column name to the output when you're using FOR XML, and your Powershell script needs to know the column name.
First I'd change your SQL query to something like:
$Query = #'
SELECT (
SELECT *
FROM TableName
WHERE Condition = 1
FOR XML PATH ('Foo'), ROOT ('Bar')
) AS XmlOutput
'#
That will give your query output a fixed name.
Then you do this:
$DataSet.Tables[0].XmlOutput | Add-Content $FileName -Encoding utf8
I would avoid Out-File in favor of Set-Content because in some versions of Powershell it can have issues with truncating long lines.
Also, be aware that utf8 encoding includes a byte order mark. That's valid according to the Unicode spec, but lots of systems don't like it especially on Linux. If you're on Powershell v6+ you can specify -Encoding utf8nobom, but Windows Powershell v5.1 and earlier don't have that option.

Related

How may I export the output of SQL query in Excel fetched through PowerShell?

I am using this power-shell script to fetch the versions of all SQL Servers in a list.
How may I export the result columns (only query output not error messages) into excel and send to email after the script is run?
Can someone help me add the required script please?
Import-Module SQLPS -DisableNameChecking
$ServerInstences = Get-Content "D:\DBA\All_Server_monitoring\ServerList.txt"
$SQLQuery = #"
Select ##Servername 'Server Name' ,##version 'Version'
"#
$DBName = "master"
$ServerInstences |
ForEach-Object {
$ServerObject = New-Object -TypeName Microsoft.SqlServer.Management.Smo.Server -ArgumentList $_
Invoke-Sqlcmd -ServerInstance $_ -Database $DBName -Query $SQLQuery
}
The easiest way to export data to a csv file is by using Export-CSV which takes an input object (or object array) and a path and can fill out the csv file from that object/array. For you, it would look like this:
$results = Invoke-Sqlcmd -ServerInstance $_ -Database $DBName -Query $SQLQuery
New-Item -Path "MYPATH.csv"
Export-CSV -Path "MYPATH.csv" -InputObject $results -Append
CSV files are versatile and can be opened with the most lightweight text editors. They also can be easily emailed and opened with Excel.

Join SQL query Results and Get-ChildItem Results

Background: I have a directory with a number of files that are imported to SQL server.
Task: Creating a PowerShell script which will pick up files within this directory and use the filenames as in the SQL query.
Ultimate objective: To display SQL results besides the filenames but the resultset being displayed should also show files having no entries in SQL server. Something like RIGHT JOIN in SQL server queries.
Powershell Code
$files = Get-ChildItem -Recurse -Force $filePath -ErrorAction SilentlyContinue | Where-Object { ($_.PSIsContainer -eq $false) } | Select-Object Name
$Server = "Loadv1"
$DB = "LoadDB"
$dbResults = #()
ForEach ($file in $files)
{
$fileName = $file.name
write-host $fileName
if($fileName.Length -gt 1)
{
$Query = "
SELECT FileName,CurrentStatus
FROM LogStatus
WHERE FileName LIKE '$fileName%'
"
# Write-host $Query
}
$dbResults += Invoke-Sqlcmd -ServerInstance $Server -Database $DB -Query $Query
}
$dispResults = $dbResults,$file
$dispResults | Format-Table -autosize
Work done so far: I have been able to fetch the file names using Get-ChildItem and loop them to get the query results. However, the result I am currently getting does not show the files that don't have corresponding entry in SQL server table
Current Result
OperationalLanding20150622061502.dat
OperationalLandingAudit20150622061502.dat
OperativeThird_Party_System20150616090701.dat
FileName CurrentStatus
OperationalLandingAudit20150622061502.dat SSIS Package Complete
OperativeThird_Party_System20150616090701.dat SSIS Package Complete
Expected Result
OperationalLanding20150622061502.dat
OperationalLandingAudit20150622061502.dat
OperativeThird_Party_System20150616090701.dat
FileName CurrentStatus
OperationalLanding20150622061502.dat NULL
OperationalLandingAudit20150622061502.dat SSIS Package Complete
OperativeThird_Party_System20150616090701.dat SSIS Package Complete
Hoping I was able to explain my requirement above.
OK so if the SQL query does not have results then NULL is returned and, in essence, nothing is added to the $dbResults array. Instead lets append the results to a custom object. I don't know what PowerShell version you have so I needed to do something that I know should work. I also don't use the SQL cmdlets much so I had to guess for some of this.
$files = Get-ChildItem -Recurse -Force $filePath -ErrorAction SilentlyContinue |
Where-Object {$_.PSIsContainer -eq $false -and $_.Length -gt 1} |
Select-Object -ExpandProperty Name
$Server = "Loadv1"
$DB = "LoadDB"
$files | ForEach-Object{
write-host $_
$Query = "
SELECT FileName,CurrentStatus
FROM LogStatus
WHERE FileName LIKE '$_%'
"
$Results = Invoke-Sqlcmd -ServerInstance $Server -Database $DB -Query $Query
$props = #{Name = $_}
If($Results){
$props.CurrentStatus = $Results.CurrentStatus
} Else {
$props.CurrentStatus = "Null"
}
New-Object -TypeName PSCustomObject -Property $props
} | Format-Table -autosize
What this does is create a custom object that contains the results of the sql query (Which I did not change for reasons stated above). If there are no results returned we use the string "null" as a filler.
I cleaned up how you generated the $files variable by making is a simple string array with -Expand and moved the length condition there as well.
You should now have all the expected results. I say should since I am assuming what the return object looks like.
$Query = "
SELECT isNull(A.FileName, b.FileName) FileName,ISNULL(A.CurrentStatus,B.CurrentStatus) CurrentStatus
FROM LogStatus A
Right JOIN (SELECT '$filename' FileName,NULL CurrentStatus) B
ON a.Filename like '$filename%'
"
This should pad out the filenames for you. A little tough to prototype since it's in powershell but I might be able to come up with a sql fiddle to prove it.
EDIT
Answer edited, with sql fiddle:
http://sqlfiddle.com/#!3/12b43/9
Obviously, since you're in a cursor, we can only prove one query at a time.

In powershell, how can I append one table to another and export to CSV?

I'm relatively new to Powershell, but here's what I'm trying to do:
I need to make multiple SQL queries and store the results to all of the queries in the same csv. With Powershell 3.0 I could use Export-CSV -Append, but unfortunately I need to use Powershell 2.0. Here is what I have right now.
#Connection Strings
$Database = "DB"
$Server = "localhost"
#Export File
$AttachmentPath = "C:\Users\Administrator\Desktop\SQLData.csv"
# Connect to SQL and query data, extract data to SQL Adapter
$SqlQuery = "select * from DB.dbo.DB1"
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source=$Server;Initial Catalog=$Database;Integrated Security = True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$nRecs = $SqlAdapter.Fill($DataSet)
$nRecs | Out-Null
#Populate Hash Table
$objTable = $DataSet.Tables[0]
#Export Hash Table to CSV File
$objTable | Export-CSV $AttachmentPath
Write-Output "REPORT: Successfully created ${AttachmentPath}"
This will successfully create and export my table into a csv file that looks like this:
Table 1
----------------
col1(int) col2(string) col3(string)
col1(int) col2(string) col3(string)
but now I want to run a different query the exact same way
# Connect to SQL and query data, extract data to SQL Adapter
$SqlQuery = "select * from DB.dbo.DB2"
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source=$Server;Initial Catalog=$Database;Integrated Security = True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$nRecs = $SqlAdapter.Fill($DataSet)
$nRecs | Out-Null
#Populate Hash Table
$objTable = $DataSet.Tables[0]
#IN POWERSHELL 3.0 THIS WOULD WORK
$objTable | Export-CSV $AttachmentPath -Append
and append it onto the first table. The final CSV would look like this:
Table 1
----------------
col1(int) col2(string) col3(string)
col1(int) col2(string) col3(string)
Table 2
----------------
col1(string) col2(int) col3(int)
col1(string) col2(int) col3(int)
I can't find any way to do this, any help is appreciated!
EDIT
Found a workaround by exporting the second query to a csv,
$objTable | Export-CSV $AttachmentPath
Then I used the Get-Content and Add-Content cmdlets to append to the first csv. It's hacky but it works. If you have anything better let me know!
$file2=Get-Content $AttachmentPath
Add-Content "SQLData.csv" "`n"
Add-Content "SQLData.csv" $file1
Why can't you use UNION in your query like below to get a merged data from both table
$SqlQuery = "select * from DB.dbo.DB1 UNION select * from DB.dbo.DB2"
Then you can use Export-CSV filename.csv commandlet to export the data to a CSV file (OR) see this thread How to export data to CSV in PowerShell?
While Union might not be acceptable for all situations, it is possible to merge two queries... just cast the output of numbers and dates to varchar/char.
SELECT Cast(col1(int) as varchar(7)) as col1, col2(string) as col2,
col3(string)
FROM [Table 1]
UNION ALL SELECT col1(string), Cast(col2(int) as varchar(7)),
Cast(col3(int) as varchar(7))
Just make sure you aren't truncating any data and you can mash em together.
Of course with 15 tables joining them in powershell would seem easier to maintain.

Remove column header from SQL Server query result

I want to remove column header from SQL Server query output. I did the search but not found any solution. I have a query eg.
select cc.DepartmentID , cc.Name from HumanResources.Department cc
When I run this query I am getting output like this.
ID Name
12 Document Control
1 Engineering
16 Executive
14 Facilities and Maintenance
10 Finance
9 Human Resources
I want to remove ID and Name (Column Header) from the output in SQL Server.
I will run this query by script to generate csv file.
Edit:
When i run the query by script i got the csv file as output and it look like this.
#TYPE System.Data.DataRow
ID Name
Update:
I am putting powershell script.
$Database = "temp"
$Server = "localhost"
$AttachmentPath = "output.csv"
# Connect to SQL and query data, extract data to SQL Adapter
$SqlQuery = "select cc.DepartmentID , cc.Name from HumanResources.Department cc"
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source=$Server;Initial Catalog=$Database;Integrated Security = True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$nRecs = $SqlAdapter.Fill($DataSet)
$nRecs | Out-Null
#Populate Hash Table
$objTable = $DataSet.Tables[0]
#Export Hash Table to CSV File
$objTable | Export-CSV $AttachmentPath
I want to remove column header from output.
In SSMS Under Tools/Options/Query Results/SQL Server/Results to Text there is a checkbox to 'Include column headers in the result set'. Similar for results to grid.
If you are using sqlcmd via powershell you can use /h-1 to disable the headers.
This setting corresponds to the environment variable SQLCMDHEADERS.
Tips and Tricks
Use a value of -1 to specify that no headers should be
printed. If -1 is supplied, there must be no space between the
parameter and the setting, that is, -h-1. Otherwise, SQLCMD interprets
it as a separate option and fails.
Example (modified from [TechNet])1:
sqlcmd -q /h-1 "SELECT * FROM AdventureWorks2012.Person.Person"
will also work with -h-1
In management studio at query window right click and select Query options. Look for Result>Text at a tree in the left and check out Include column headers in result set option. I think Hamlet Hakobyan is right, it is client that add column headers.
Replace your last line $objTable | Export-CSV $AttachmentPath with
$objTable | ConvertTo-Csv -NoTypeInformation | select -Skip 1 | out-file $AttachmentPath
Using the Save As option would not include the attribute (column) names.
This work correctly and column header not exists in out-file:
$workpath = "C:\myworkdir"
$InvSQLParams = #{
ServerInstance = "SQL2016"
Database = "testdb"
InputFile = "$($workpath)\selectclause.sql"
}
Invoke-Sqlcmd #InvSQLParams | ConvertTo-Csv -NoTypeInformation | select -Skip 1 | out-file "$($workpath)\result.csv"
in your script, pipe (|) the output to the "tail +3" command.
this will skip the first 2 lines of output from the SQL.
set this after connecting to database
SET HEADING OFF

XML data coming in newline using invoke-sql command in powershell

I am using a powershell script which will return an XML from a sqlquery of FOR XML EXPLICIT type.
Now, the query returned the XML in a column with header and dashed lines
XML 32776498797309
---------------------
<data>asdsafafaf<\data><value>dfsasfasfdfasdf....
dfdsfsdgregrge<\value><value>asdfasdfadfadfsda<\v
alue>
Here, some how we were able to remove header , dashed line and truncating of data.
but still the data comes in new line so if we open the XML it throws error because at some places the tag gets distributed as shown above.
Basically, new line issue is there.
We tried -width 4096
but since XML is huge it is not proper.
Please help, stuck big time.
Query used :
invoke-sqlcmd -inputFile $inputFilePath -serverinstance $dbHostName
-database $dbName -username $userName -password $password
| Format-Table -hide -Wrap -AutoSize
| Out-File -filePath $outputFilePath -width 4096
Try using set-content instead of out-file. Out-file will use the host/console formatting while set-content does not. You'll need to handle outputting XML to string
invoke-sqlcmd ....| select -expandproperty XML | set-content -path $outputFilePath
Edited. Added full working example:
#Contents of my test inputfile.sql:
SELECT CAST ('<EVENT_INSTANCE>
<EventType>CREATE_TABLE</EventType>
<PostTime>2011-04-26T15:05:21.333</PostTime>
<SPID>56</SPID>
<ServerName>Z001\SQL1</ServerName>
<LoginName>Contoso\u00</LoginName>
<UserName>dbo</UserName>
<DatabaseName>AdventureWorksDW2008R2</DatabaseName>
<SchemaName>dbo</SchemaName>
<ObjectName>AdventureWorksDWBuildVersion</ObjectName>
<ObjectType>TABLE</ObjectType>
<TSQLCommand>
<SetOptions ANSI_NULLS="ON" ANSI_NULL_DEFAULT="ON" ANSI_PADDING="ON" QUOTED_IDENTIFIER="ON" ENCRYPTED="FALSE" />
<CommandText>CREATE TABLE [dbo].[AdventureWorksDWBuildVersion] (
[DBVersion] [nvarchar] (50) NULL,
[VersionDate] [datetime] NULL
) ON [PRIMARY];
</CommandText>
</TSQLCommand>
</EVENT_INSTANCE>' AS XML)
$inputFilePath = "C:\Temp\inputfile.sql"
$dbHostName = "$env:computername\sql1"
$outputFilePath = "C:\Temp\output.txt"
#Notice Extra Line Breaks
invoke-sqlcmd -inputFile $inputFilePath -serverinstance $dbHostName |
Format-Table -hide -Wrap -AutoSize |
Out-File -filePath $outputFilePath -width 4096
#No Extra line breaks using set-content
invoke-sqlcmd -inputFile $inputFilePath -serverinstance $dbHostName | select -expandproperty Column1 | set-content -Path $outputFilePath
Use -MaxCharLength parameter of Invoke-SQLCMD command. By default it 4000. it will truncate big XMLs.
See Invoke-SqlCmd doesn't return long string?

Resources