If I want to insert the values in the array in one connection, how can i modify the code below? So far i get the error "There is already an open DataReader associated with this Command which must be closed first" unless I put the $Connection.Open() and $Connection.Close() inside the for loop, which will cost me the speed of a single connection.
$list = 'aaa','bbb','cccc','ddddd','eeeee','ffff'
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$Server';database='$Database';trusted_connection=true;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
foreach($i in $list) {
$sql ="if not exists (select 1 from [bfs] where [key] = '$i' )
begin
insert bfs
select '$i'
end
"
$Command.CommandText = $sql
$Command.ExecuteReader()
}
$Connection.Close()
Try using ExecuteNonQuery() instead of ExecuteReader(). ExecuteNonQuery() does not build a DataReader.
...
$Command.CommandText = $sql
$Command.ExecuteNonQuery()
...
I am trying to run a SQL from Power Shell(which is on my windows 7 64 bit desktop) and the remote database host is MS SQL Server 2012.
I am running SQL1 by calling function Get-ODBC-Data which will give me single column of type string. It can have unknown number of rows (up to 20). Then I am using each of these column values as parameter ($var1) to the second function Get-ODBC-Data-Count. The SQL2 in this function Get-ODBC-Data-Count will give me count using $var1 in where clause.
The code is:
function Get-ODBC-Data{
param(
[string]$query=$('
SELECT col3
FROM [master].[sys].[table_name]'),
[string]$username='db_user_name',
[string]$password='db_password'
)
$conn = New-Object System.Data.Odbc.OdbcConnection
$conn.ConnectionString = "DRIVER={SQL Server};Server=123.456.78.90;Initial Catalog=master;Uid=$username;Pwd=$password;"
$conn.open()
$cmd = New-object System.Data.Odbc.OdbcCommand($query,$conn)
$ds = New-Object system.Data.DataSet
(New-Object system.Data.odbc.odbcDataAdapter($cmd)).fill($ds) | out-null
$conn.close()
$ds.Tables[0]
}
function Get-ODBC-Data-Count{
[parameter(Mandatory=$true)][string]$var1,
param(
[string]$query=$('
SELECT COUNT(*)
FROM [master].[sys].[table_name]
WHERE col2 = '$($var1)'
;
'),
[string]$username='db_user_name',
[string]$password='db_password'
)
$conn = New-Object System.Data.Odbc.OdbcConnection
$conn.ConnectionString = "DRIVER={SQL Server};Server=123.456.78.90;Initial Catalog=master;Uid=$username;Pwd=$password;"
$conn.open()
$cmd = New-object System.Data.Odbc.OdbcCommand($query,$conn)
$ds = New-Object system.Data.DataSet
(New-Object system.Data.odbc.odbcDataAdapter($cmd)).fill($ds) | out-null
$conn.close()
$ds.Tables[0]
}
$result = Get-ODBC-Data
$count_1 = Get-ODBC-Data-Count -var1 $result[0][0]
Write-Host "`$count_1[0]:" $count_1[0];
Write-Host "Message: Count of " $result[0][0] " is" $count_1[0];
$count_2 = Get-ODBC-Data-Count -var1 $result[1][0]
Write-Host "`$count_2:" $count_2[0];
Write-Host "Message: Count of " $result[1][0] " is" $count_2[0];
$count_3 = Get-ODBC-Data-Count -var1 $result[2][0]
Write-Host "`$count_3:" $count_3[0];
Write-Host "Message: Count of " $result[2][0] " is" $count_3[0];
This code works if I know number of rows in the result of SQL1.
My Question: How can I modify this code so unknown number of rows in result of SQL1 will be handled and I can call function Get-ODBC-Data-Count for each row of SQL1?
There are a lot of issues here.
You're building SQL strings. Don't do this! Use SQL parameters instead!
You're repeating a lot of code unessescarily.
You're using Data Tables, which I would avoid, at least in powershell.
You're not re-using the database connection.
Always try really hard to avoid loops with a query inside when working with SQL. Try and think if you can rewrite the SQL instead.
Try this SQL:
SELECT
col2,
COUNT(<thePrimaryKeyColumnOfTheTable>)
FROM [master].[sys].[table_name]
GROUP BY col2
That should give you the count of all the different values of col2.
I have this powershell script that would work if my DEST table ONLY had the columns listed in the select from my SOURCE server, but the DEST table has more. I haven't been able to find anything that gives examples on how to specify the columns from my dest table I want to insert into. Note that the SourceServer and DestServer are not linked servers.
Param (
#[parameter(Mandatory = $true)]
[string] $SrcServer = "SourceServer",
[parameter(Mandatory = $true)]
[string] $SrcDatabase = "SourceDb",
#[parameter(Mandatory = $true)]
[string] $SrcTable = "stage.InternalNotes",
#[parameter(Mandatory = $true)]
[string] $DestServer = "DestServer",
#[parameter(Mandatory = $true)]
[string] $DestDatabase = "DestDb",
[parameter(Mandatory = $true)]
[string] $DestTable = "dbo.InternalNotes",
)
Function ConnectionString([string] $ServerName, [string] $DbName)
{
"Data Source=$ServerName;Initial Catalog=$DbName;Integrated Security=True;User ID=$UID;Password=$PWD;"
}
$SrcConnStr = ConnectionString $SrcServer $SrcDatabase
$SrcConn = New-Object System.Data.SqlClient.SQLConnection($SrcConnStr)
$CmdText = "SELECT
ino.UserId
,ino.StoreId
,ino.PostedById
,ino.DatePosted
,ino.NoteSubject
,ino.NoteText
,ino.NoteType
,ino.Classify
,ino.CreatedBy
,ino.CreatedUtc
,IsReadOnly = 0
FROM
stage.InternalNotes AS ino
"
$SqlCommand = New-Object system.Data.SqlClient.SqlCommand($CmdText, $SrcConn)
$SrcConn.Open()
[System.Data.SqlClient.SqlDataReader] $SqlReader = $SqlCommand.ExecuteReader()
Try
{
$DestConnStr = ConnectionString $DestServer $DestDatabase
$bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($DestConnStr, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity)
$bulkCopy.DestinationTableName = $DestTable
$bulkCopy.WriteToServer($sqlReader)
}
Catch [System.Exception]
{
$ex = $_.Exception
Write-Host $ex.Message
}
Finally
{
Write-Host "Table $SrcTable in $SrcDatabase database on $SrcServer has been copied to table $DestTable in $DestDatabase database on $DestServer"
$SqlReader.close()
$SrcConn.Close()
$SrcConn.Dispose()
$bulkCopy.Close()
}
Essentially, I need to be able to do this:
INSERT INTO dbo.InternalNotes --DEST Server table
(
userID
,StoreID
,PostedByID
,DatePosted
,NoteSubject
,NoteText
,NoteType
,Classify
,CreatedBy
,CreatedDateUTC
,IsReadOnly
)
SELECT
ino.UserId
,ino.StoreId
,ino.PostedById
,ino.DatePosted
,ino.NoteSubject
,ino.NoteText
,ino.NoteType
,ino.Classify
,ino.CreatedBy
,ino.CreatedUtc
,IsReadOnly = 0
FROM
stage.InternalNotes AS ino --SOURCE Server table
Edits after getting everything to work based on the accepted answer:
For some reason it didn't like the line:
$bulkCopy = New-Object -TypeName Data.SqlClient.SqlBulkCopy -ArgumentList $DestSqlConnection, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity, $DestSqlTransaction;
It gave the error:
Cannot convert argument "1", with value:
"[System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity", for
"SqlBulkCopy" to type "System.Data.SqlClient.SqlBulkCopyOptions":
"Cannot convert value
"[System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity" to type
"System.Data.SqlClient.SqlBulkCopyOptions". Error: "Unable to match
the identifier name
[System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity to a valid
enumerator name. Specify one of the following enumerator names and try
again: Default, KeepIdentity, CheckConstraints, TableLock, KeepNulls,
FireTriggers, UseInternalTransaction,
AllowEncryptedValueModifications""
So Instead I changed it to this, and everything worked:
$bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($DestSqlConnection, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity,$DestSqlTransaction)
To do manual column mapping, you need to populate SqlBulkCopy.ColumnMappings. If you don't specify the mapping, then as far as I know SqlBulkCopy will assume the first column in the select list or DataRow goes into the first ordinal column of the destination table.
For example:
$bulkCopy.DestinationTableName = $DestTable;
$bulkCopy.ColumnMappings.Add('sourceColumn1','destinationColumn1');
$bulkCopy.ColumnMappings.Add('sourceColumn2','destinationColumn2');
$bulkCopy.ColumnMappings.Add('sourceColumn3','destinationColumn3');
$bulkCopy.ColumnMappings.Add('sourceColumn4','destinationColumn4');
$bulkCopy.ColumnMappings.Add('sourceColumn5','destinationColumn5');
However, there's a number of other issues with your script.
Your connection string authentication section is nonsense:
`Integrated Security=True; User ID=$UID; Password=$PWD;`
Integrated Security=True says, "Use passthrough Windows authentication with currently logged on user." User ID=$UID; Password=$PWD; says, "Use SQL authentication with the specified username and password." You can't do both.
You should specify only one or the other.
$SqlCommand = New-Object system.Data.SqlClient.SqlCommand($CmdText, $SrcConn)
[...]
$bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($DestConnStr, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity)
I may be wrong, but I'm pretty sure you're trying to pass two variables as one argument here. Just like with your ConnectionString function, I don't think you don't want parentheses here. In any case it's syntactically confusing. Do this instead:
$SqlCommand = New-Object -TypeName System.Data.SqlClient.SqlCommand -ArgumentList $CmdText, $SrcConn
[...]
$bulkCopy = New-Object -TypeName Data.SqlClient.SqlBulkCopy -ArgumentList $DestConnStr, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity
Speaking of that last one, I have another issue with it. SqlBulkCopy is powerful, but you really have to hold it's hand. By default, SqlBulkCopy doesn't run with any transaction benefits. That means that if it errors in the middle, well, too bad, your data has been partially updated. You can enable internal transactions, but then only the most recent batch of the inserts will be rolled back. You really need to manage your own transaction to get an all-or-nothing result.
So you'll end up with something like this:
Try {
$DestConnStr = ConnectionString $DestServer $DestDatabase
# We have to open the connection before we can create the transaction
$DestSqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $DestConnStr;
$DestSqlConnection.Open();
$DestSqlTransaction = $DestSqlConnection.BeginTransaction();
$bulkCopy = New-Object -TypeName Data.SqlClient.SqlBulkCopy -ArgumentList $DestSqlConnection, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity, $DestSqlTransaction;
$bulkCopy.DestinationTableName = $DestTable
$bulkCopy.ColumnMappings.Add('sourceColumn1','destinationColumn1');
$bulkCopy.ColumnMappings.Add('sourceColumn2','destinationColumn2');
$bulkCopy.ColumnMappings.Add('sourceColumn3','destinationColumn3');
$bulkCopy.ColumnMappings.Add('sourceColumn4','destinationColumn4');
$bulkCopy.ColumnMappings.Add('sourceColumn5','destinationColumn5');
Try {
$bulkCopy.WriteToServer($sqlReader)
# Commit on success
$DestSqlTransaction.Commit();
}
Catch {
# Rollback on error
$DestSqlTransaction.Rollback();
# Rethrow the error to the outer catch block
throw ($_);
}
}
Catch [System.Exception] {
$ex = $_.Exception
Write-Host $ex.Message
}
Finally {
[...]
}
I'd probably rewrite the above more because I don't like nested try blocks, but for a quick and dirty rewrite this will work. I don't think you'll run into any problems with distributed transaction problems doing this, but I may be wrong. I tend to use SSIS or linked servers when I need this sort of data pump.
I have a large CSV file with 5 columns, the first row being the title of the columns. I'm trying to pass the values to a datatable using powershell that I then pass to a table-value parameter to load the data into my SQL Server instance. My SQL Server table has already been created with the 5 columns, but I am running into issues.
cls
#CSV variables
$csvfile = "C:\Students\test.csv"
$csvdelimiter = ","
$firstRowColumns = $true
#creating the datatable
$dt = New-Object System.Data.Datatable
$reader = New-Object System.IO.StreamReader $csvfile
$columns = (Get-Content $csvfile -First 1).Split($csvdelimiter)
foreach ($column in $columns)
{
if ($firstRowColumns -eq $true)
{
[void]$dt.Columns.Add($column)
$reader.ReadLine()
} else { [void]$dt.Columns.Add() }
}
# Read in the data, line by line
while (($line = $reader.ReadLine()) -ne $null)
{
[void]$dt.Rows.Add($line.Split($csvdelimiter))
}
function ExecSproc
{
param ($Conn, $Sproc, $Parameters=#{})
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandType = [System.Data.CommandType]::StoredProcedure
$SqlCmd.Connection = $Conn
$SqlCmd.CommandText = $Sproc
foreach($p in $Parameters.Keys){
[Void] $SqlCmd.Parameters.AddWithValue("#$p",$Parameters[$p])
}
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter($SqlCmd)
$DataSet = New-Object System.Data.DataSet
[Void] $SqlAdapter.Fill($DataSet)
$SqlConnection.Close()
return $DataSet.Tables[0]
}
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "server='.';database='ActiveStudents';trusted_connection=true;"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
ExecSproc -Conn $SqlConnection -Sproc spInsertStudents -Parameters #{TVP = $dt}
These are the errors being thrown:
Exception calling "Add" with "1" argument(s): "Input array is longer than the number of columns in this table."
At C:\Scripts\ActiveStudentInsert.ps1:24 char:2
Exception calling "Fill" with "1" argument(s): "Conversion failed when converting the nvarchar value '' to data type int.
The data for table-valued parameter "#TVP" doesn't conform to the table type of the parameter. SQL Server error is: 245, state: 1
The statement has been terminated."
Chad Miller made a great script in the Script Gallery that outputs DataTables in a very similar way to the one you are looking to do.
If you used his functions, then it would look something like this:
$dt = Import-Csv $csvfile | Out-DataTable
If you have a really large amount of data, I would recommend you check out the SQL BCP command-line utility.
I have a Powershell Function that is being used to run multiple queries in SQL and export as CSVs. Each of these queries relies on a date variable. Is there a way to pass this date variable from Powershell into these SQL Scripts (not stored procedures) using my current setup? Any help is much appreciated!
Function Run-Query
{
param([string[]]$queries,[string[]]$sheetnames)
Begin
{
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database = $Database; User ID = $uid; Password = $pwd;"
Write-host "Connection to database successful."
}#End Begin
Process
{
# Loop through each query
For($i = 0; $i -lt $queries.count; $i++)
{
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
# Use the current index ($i) to get the query
$SqlCmd.CommandText = $queries[$i]
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
# Use the current index ($i) to get the sheetname for the CSV
$DataSet.Tables[0] #| Export-Csv -NoTypeInformation -Path "C:\Users\mbaron\Downloads\$($sheetnames[$i]).csv"
}
}#End Process
End
{
$SqlConnection.Close()
}
}#End function run-query.
You could add a marker in your queries where the data is being used, then do a replace with the relevant date, e.g.:
cls
$date = '1/1/2016'
$query = 'some $$marker$$ script'
$query = $query.replace('$$marker$$', $date )
$query