I have a task to save the results of a SQL Server query into a .csv file. After some googling I decided to use PowerShell. I found a script, modified it a bit, it works and almost all is ok.
$server = "server"
$database = "database"
$query = "SELECT * from et_thanks"
$tod = Get-Date;
$file = "{0:yyyyMMdd}_go.csv" -f $tod;
$extractFile = #"
\\info\export_files\$file
"#
$connectionTemplate = "Data Source={0};Integrated Security=SSPI;Initial Catalog={1};"
$connectionString = [string]::Format($connectionTemplate, $server, $database)
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$command = New-Object System.Data.SqlClient.SqlCommand
$command.CommandText = $query
$command.Connection = $connection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $command
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$connection.Close()
$DataSet.Tables[0] | Export-Csv -Force -Delimiter ";" $extractFile
But I have 2 problems which I can't solve:
When I open the .csv file I see columns headers and commented string on first line:
#TYPE System.Data.DataRow
"ob_no","c_name","c_visible","c_fp","e_from","e_to"
"436439","09.09.2013 11:29:08","0","","10937","260153"
How can I get rid of it?
All values are surrounded with quotes. Is it possible to modify script not to use it while exporting? Autoreplace isn't good idea, cause there is a possibility that quote symbol can be found in sql data.
I tried to find answers in documentation (http://ss64.com/ps/export-csv.html) but with no success.
You might run in to trouble removing the quotes, but if that's what you really want then the following should achieve it.
-NoTypeInformation will remove the additional type information you are seeing.
($DataSet.Tables[0] | ConvertTo-Csv -Delimiter ";" -NoTypeInformation) -replace "`"", "" | `
Out-File -Force $extractFile
This uses convertto-csv to convert to a string representation of the csv followed by replacing all instances of " with nothing and the final string is piped to Out-File.
...and, to get rid of the header record, if you first convert the data to csv (Convert-Csv), then pipe those results to Select to skip the 1st record:
($DataSet.Tables[0] | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation ) -Replace "`"","" | Select -skip 1 | Out-File blahblahblah...
Agreed export-csv isn't the best tool for the job. I would use sqlcmd.exe or bcp.exe provided SQL Server command-lines tools are installed. You could also build a simple routine to create a CSV from a datatable:
$result = new-Object text.stringbuilder
$dt = $DataSet.Tables[0]
foreach ($dr in $dt.Rows) {
for ($i = 0; $i -lt $dt.Columns.Count; $i++) {
$null = $result.Append($($dr[$i]).ToString())
$null = $result.Append($(if ($i -eq $dt.Columns.Count - 1) {"`n" } else { ","} ))
}
}
$result.ToString()
Related
I'm trying to execute an SQL query with few select statements, that returns multiple tables as a result. The problem is that I can't find a way to read and use the tables separately.
Expected results:
Actual results: (it is printed row by row)
Purpose: I've made a script that creates an empty excel file with multiple sheets and each of the sheets will be used to contain each resultset of the query.
The only thing left is to put the needed text into the sheets. Here is my code for that part only:
$ConnectionString = "Data Source=...;Initial Catalog=...;User Id=...;Password=..."
$DBServerName = $ConnectionString.split('=')[1].split(';')[0]
$DBName = $ConnectionString.split('=')[2].split(';')[0]
$DBUser = $ConnectionString.split('=')[3].split(';')[0]
$DBPassword = $ConnectionString.split('=')[4].split(';')[0]
$CurrentFilePath = "C:\SQLqueryWithManyResultsets.sql"
$query = Get-Content -literalPath $CurrentFilePath | Out-String #getting the query string from file
$resultTables = Invoke-Sqlcmd -Query $query -ServerInstance $DBServerName -Database $DBName -DisableVariables -Password $DBPassword -Username $DBUser -ErrorAction Stop
foreach ($result in $resultTables) {
$result | Format-Table #where the magic happens
}
I've made a lot of research, but I cannot find a proper way to store and read the tables the way i need.
Try this:
Clear-Host;
$objConnection = New-Object System.Data.SqlClient.SqlConnection;
$objConnection.ConnectionString = "...";
$ObjCmd = New-Object System.Data.SqlClient.SqlCommand;
$ObjCmd.CommandText = "...";
$ObjCmd.Connection = $objConnection;
$ObjCmd.CommandTimeout = 0;
$objAdapter = New-Object System.Data.SqlClient.SqlDataAdapter;
$objAdapter.SelectCommand = $ObjCmd;
$objDataSet = New-Object System.Data.DataSet;
$objAdapter.Fill($objDataSet) | Out-Null;
for ($i=0; $i -lt $objDataSet.Tables.Count; $i++) {
Write-Host ($objDataSet.Tables[$i] | Format-Table | Out-String);
}
$query = $null;
$objDataSet = $null;
$objConnection.Close();
$objConnection = $null;
I have a large CSV file with 5 columns, the first row being the title of the columns. I'm trying to pass the values to a datatable using powershell that I then pass to a table-value parameter to load the data into my SQL Server instance. My SQL Server table has already been created with the 5 columns, but I am running into issues.
cls
#CSV variables
$csvfile = "C:\Students\test.csv"
$csvdelimiter = ","
$firstRowColumns = $true
#creating the datatable
$dt = New-Object System.Data.Datatable
$reader = New-Object System.IO.StreamReader $csvfile
$columns = (Get-Content $csvfile -First 1).Split($csvdelimiter)
foreach ($column in $columns)
{
if ($firstRowColumns -eq $true)
{
[void]$dt.Columns.Add($column)
$reader.ReadLine()
} else { [void]$dt.Columns.Add() }
}
# Read in the data, line by line
while (($line = $reader.ReadLine()) -ne $null)
{
[void]$dt.Rows.Add($line.Split($csvdelimiter))
}
function ExecSproc
{
param ($Conn, $Sproc, $Parameters=#{})
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandType = [System.Data.CommandType]::StoredProcedure
$SqlCmd.Connection = $Conn
$SqlCmd.CommandText = $Sproc
foreach($p in $Parameters.Keys){
[Void] $SqlCmd.Parameters.AddWithValue("#$p",$Parameters[$p])
}
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter($SqlCmd)
$DataSet = New-Object System.Data.DataSet
[Void] $SqlAdapter.Fill($DataSet)
$SqlConnection.Close()
return $DataSet.Tables[0]
}
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "server='.';database='ActiveStudents';trusted_connection=true;"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
ExecSproc -Conn $SqlConnection -Sproc spInsertStudents -Parameters #{TVP = $dt}
These are the errors being thrown:
Exception calling "Add" with "1" argument(s): "Input array is longer than the number of columns in this table."
At C:\Scripts\ActiveStudentInsert.ps1:24 char:2
Exception calling "Fill" with "1" argument(s): "Conversion failed when converting the nvarchar value '' to data type int.
The data for table-valued parameter "#TVP" doesn't conform to the table type of the parameter. SQL Server error is: 245, state: 1
The statement has been terminated."
Chad Miller made a great script in the Script Gallery that outputs DataTables in a very similar way to the one you are looking to do.
If you used his functions, then it would look something like this:
$dt = Import-Csv $csvfile | Out-DataTable
If you have a really large amount of data, I would recommend you check out the SQL BCP command-line utility.
I am trying to import a csv file where it doesn't import any values that are listed in an array declared above the import line. The array is made up of certain values that are pulled out of a database and I wan't to import all rows in the csv file that the txnID column values do not match the values in the array however I am having trouble trying to loop through my array.
I am new to using powershell and maybe I am not even implementing the array correctly but I haven't been able to find anything about import-csv Filename |Where column -notmatch $array
$Database = 'Database'
$Server = "Server"
$SqlQuery = 'SELECT DISTINCT WebOrderNumber FROM tbOrders
WHERE WebOrderNumber IS NOT NULL AND Len(WebOrderNumber)>8'
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source=Datasource;Initial Catalog=Database;User ID=ID;Password=Pass;Integrated Security=False;"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
$SqlConnection.Open()
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$Reader = $SqlCmd.ExecuteReader()
while ($Reader.Read()) {
#write-Output($Reader.GetValue($0))
$Key = $Reader.GetValue($0)
$table += $Key
}
foreach ($Row in $table){
write-output($Row)
$CSVFile = (import-csv "C:\Users\Office-Admin\Documents\Complete Sales Orders.csv") |where {$_.txnID -ne $Row} | select txnID, FirstName, LastName, Cust_Name, mc_Shipping, Payment_Gross, address_street, Address_Zip, quantity, item_name, item_number, payer_email, address_city, address_state, address_country, address_name, Shipping_Method, mc_gross
}
$CSVFile | export-csv "C:\Users\Office-Admin\Documents\Sales Order Import List.csv" -notypeinformation
remove-item variable:table
#Send SMTP Message
$SqlConnection.Close()
I've updated my code slightly however the problem still persists. I'm realizing that I believe with the code now, everytime I loop through and import, the previous condition in the where is forgotten so the only value that is not imported in the end is the last $Row value but I need all of the values in $table to be excluded and I don't know how I can do this.
Something list this should work. The main problem is you are over writing your csv every loop.
$table = import-csv file1.csv | % {$_ID} #gets array of just the ID values
$CSVFile = Import-csv file2.csv | where{$table -notcontains $_.ID} | export-csv output.csv -notypeinformation
To show you how this works I created to files as an example:
File 1: CSV with IDs:
ID,Stuff
123,alittlestuff
234,morestuff
345,evenmore
456,alotmore
567,somemore
678,notsomuch
789,tonesofstuff
File 2: csv with ID and stuff:
ID,stuff
123,hello
ghf,world
234,test
lkj,this
after running the code the only rows that get output are:
ID,Stuff
ghf,world
lkj,this
So I think to fit it into your code use this:
$filter = $table | %{$_.txnID}
$CSVFile = (import-csv "C:\Users\Office-Admin\Documents\Complete Sales Orders.csv") | where{$filter -notcontains $_.txnID} || export-csv "C:\Users\Office-Admin\Documents\Sales Order Import List.csv" -notypeinformation
I'm seeing some odd behavior. On my machine, PowerShell returns the recordset and I can iterate through the records no problem. On my co-worker's machine (who has access to the file share that I need to copy the files from) is getting a record count returned instead the actual records. I must be missing something easy. Any idea why I'm seeing this different behavior?
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = server; Database = db; Integrated Security = True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = "SELECT fileName from SomeTable"
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$Table = new-object data.datatable
$Table = $DataSet.tables[0]
$SqlConnection.Close()
function Out-FileForce {
PARAM($path)
PROCESS
{
if(Test-Path $path)
{
Out-File -inputObject $_ -append -filepath $path
}
else
{
new-item -force -path $path -value $_ -type file
}
}
}
foreach ($Row in $Table.Rows)
{
$fullPath = $Row.FullFilePathWithName
$path = "\\server\folder\"
$newPath = "C:\newFolder\"
$newDestination = $fullPath -replace [regex]::Escape($path), $newPath
#Write-Output $newDestination
#Write-Output $fullPath
# recurse should force the creation of the folder structure
#Copy-Item $fullPath $newDestination -recurse
Out-FileForce $newDestination
Copy-Item $fullPath $newDestination -force
Write-Output $newDestination " done"
}
This line:-
$SqlAdapter.Fill($DataSet)
returns the row count if you would like that for later assign it to something:-
$rowCount = $SqlAdapter.Fill($DataSet)
or if you don't require it:-
[void]$SqlAdapter.Fill($DataSet)
both of the above will avoid the need to skip 1
Hope this helps
Figured it out a fix, I'm still not sure why.
$DataSetTableRows was causing the issue
Fixing the original script I posted.
Added this to the top:
$Table = new-object data.datatable
$Table = $DataSet.tables[0]
Then in my loop I used $Table.Rows
My company has a program that tracks our Employee workouts. When we had this program made, we did not think about adding the ability to Add or Remove an employee to the program.
I wrote a script in PowerShell that allows us to do this easier than in SSMS. I would like to see if anyone can help me clean it up a bit and fine tune it.
My biggest headache is this 1 or -1 that gets returned anytime we execute a function. I would also like this to ask if they are finished, then loop back or exit. Right now it just exits as soon as they are done.
<#Writes the invoker to log#>
$trandate = Get-Date
$tranuser = $env:UserName
<# Variables to open the connection to the SQL server #>
$sqlcn = New-Object System.Data.SqlClient.SqlConnection
$sqlcn.ConnectionString = "server=10.10.1.19\VTSWORKOUT;Integrated
Security=true;Database=VTSWORKOUT;"
<# Read what the user wants to do #>
$input = Read-Host "Do you want to [A]dd a New Employee, [R]emove an Employee or [E]xit?"
switch($input){
<# Stuff for adding an employee to the database #>
A{
$eid = Read-Host "What is the Employees ID number?"
$fname = Read-Host "What is the Employees first name?"
$lname = Read-Host "What is the Employees last name?"
$dept = Read-Host "What department is the Employee in?"
$pay = Read-Host "Is the Employee Salaried? [0]Yes or [1]No"
$hire = Read-Host "When was the Employee hired? Input as MM-DD-YYYY"
Out-File -FilePath "L:\Personnel\WorkoutApp\workouts.log" -Append -InputObject "On $trandate, $tranuser added Employee# $eid, $fname $lname"
$sqlcn.Open()
$sqlcmd = $sqlcn.CreateCommand()
$query = "INSERT INTO employees values (#eid,#lname,#fname,#dept,#pay,#hire)"
$sqlcmd.CommandText = $query
$sqlcmd.Parameters.AddWithValue("#eid", $eid) | Out-Null
$sqlcmd.Parameters.AddWithValue("#fname", $fname) | Out-Null
$sqlcmd.Parameters.AddWithValue("#lname", $lname) | Out-Null
$sqlcmd.Parameters.AddWithValue("#dept", $dept) | Out-Null
$sqlcmd.Parameters.AddWithValue("#pay", $pay) | Out-Null
$sqlcmd.Parameters.AddWithValue("#hire", $hire) | Out-Null
$sqlcmd.ExecuteNonQuery()
$sqlcn.Close()
}
<# Stuff for removing an employee from the database#>
R{
<#Collect reason for removal#>
$reason = Read-Host -Prompt "Why are you deleting this employee?"
$eid = Read-Host "What is the ID number of the Employee you want to remove?"
$sqlcn.Open()
$sqlcmd = $sqlcn.CreateCommand()
$query = "SELECT EmployeeID,FirstName, LastName from Employees WHERE EmployeeID = #eid"
$sqlcmd.CommandText = $query
$sqlcmd.Parameters.AddWithValue("#eid", $eid) | Out-Null
$sqlcmd.ExecuteNonQuery()
$Reader = $sqlcmd.ExecuteReader()
$arry = #()
while ($Reader.Read()) {
$row = #{}
for ($i = 0; $i -lt $reader.FieldCount; $i++)
{
$row[$reader.GetName($i)] = $reader.GetValue($i)
}
#convert hashtable into an array of PSObjects
$arry+= new-object psobject -property $row
}
$sqlcn.Close()
write-host $arry
$empResult = Read-Host "Is that the correct employee? [Y]es or [N]o"
<#If the correct employee was found, continue below.
If the wrong employee was returned, Kill Program #>
switch($empResult) {
Y{
Out-File -FilePath "L:\Personnel\WorkoutApp\workouts.log" -Append -InputObject "On $trandate, $tranuser deleted Employee $eid for the following reason: $reason"
$sqlcn.Open()
$sqlcmd = $sqlcn.CreateCommand()
$query = "DELETE FROM Employees WHERE EmployeeID = #eid"
$sqlcmd.CommandText = $query
$sqlcmd.Parameters.AddWithValue("#eid", $eid)
$sqlcmd.ExecuteNonQuery()
$adp = New-Object System.Data.SqlClient.SqlDataAdapter $sqlcmd
$data = New-Object System.Data.DataSet
$adp.fill($data) | Out-Null
$sqlcn.Close()
}
N{
Out-File -FilePath "L:\Personnel\WorkoutApp\workouts.log" -Append -InputObject "On $trandate, $tranuser tried to deleted Employee $eid. But exited the program before doing so."
Write-Host "Please restart the program. If the issue persists, please contact the IT department."
Read-Host -Prompt "Press Enter to exit"
}
}
}
<# Line to exit the program #>
E{
exit
}
}
Any thoughts on cleaning this up would be greatly appreciated.
This is off-topic, but I'll give you an answer.
Generally, you don't want to use Parameters.AddWithValue() at all, because that sends every parameter as an NVARCHAR. It's not deprecated, but it's not a good idea to use it. If you've got datetimes or other non-string parameters, you can end up with problems. It's usually preferable to use Parameters.Add():
$sqlcmd.Parameters.Add("#eid", [System.Data.SqlDbType]::Int).Value = $eid
Obviously, the datatype you use from [System.Data.SqlDbType] should match the datatype of the actual column in the database. This also has the benefit that there won't be any return value that you need to send to Out-Null or cast as [void].
This is also a mess:
$sqlcmd.ExecuteNonQuery()
$Reader = $sqlcmd.ExecuteReader()
$arry = #()
while ($Reader.Read()) {
$row = #{}
for ($i = 0; $i -lt $reader.FieldCount; $i++)
{
$row[$reader.GetName($i)] = $reader.GetValue($i)
}
#convert hashtable into an array of PSObjects
$arry+= new-object psobject -property $row
}
First, you're executing the query twice. Both ExecuteNonQuery() and ExecuteReader() will execute the query! You do that multiple times in your script.
Second, you can just do this:
$DataTable = New-Object System.Data.DataTable
$DataTable.Load($sqlcmd.ExecuteReader())
Then, if you really don't want to work with a DataTable -- they're more complex than a custom object but really not that bad -- you can do this to convert it to a generic object pretty easily:
$Data = $DataTable | ConvertTo-Csv -NoTypeInformation | ConvertFrom-Csv
This will make everything a string, though, so be sure that's what you want. You might also try this:
$Data = $DataTable | Select-Object -Property <list>
You don't want to use Select-Object * because you'll get extra properties you probably don't want.
This is also executing the query twice:
$sqlcn.Open()
$sqlcmd = $sqlcn.CreateCommand()
$query = "DELETE FROM Employees WHERE EmployeeID = #eid"
$sqlcmd.CommandText = $query
$sqlcmd.Parameters.AddWithValue("#eid", $eid)
$sqlcmd.ExecuteNonQuery()
$adp = New-Object System.Data.SqlClient.SqlDataAdapter $sqlcmd
$data = New-Object System.Data.DataSet
$adp.fill($data) | Out-Null
$sqlcn.Close()
Both $sqlcmd.ExecuteNonQuery() and $adp.fill($data) execute the query! Additionally, ExecuteNonQuery() returns the number of records affected. You could do this:
$sqlcmd.ExecuteNonQuery() | Out-Null
Or this:
[void]$sqlcmd.ExecuteNonQuery()
But what you really should do is verify that the result is what you expect. You shouldn't be getting -1 for INSERT or DELETE statements.
Learn to look up the documentation for the methods you're calling and understand what the possible return values are and why. All the .Net methods are thoroughly documented on MSDN. You can almost always find them by Googling "C# ". You'll find C# examples that can easily be converted to PowerShell, too.