SQL query for Bulk Update - sql-server

I need to update a table using text file. Currently my code works fine if I perform Get-Content from txt file and then run the SQL update query, but only in case of small data. If the size of text is too long or it contains some special characters, it throws an error as following:
Exception calling "ExecuteReader" with "0" argument(s): "Incorrect syntax near
')</td><td style=\"border:1px solid #cccccc\">#fieldValueEmpty($issue.getCustom
FieldValue($componentTypeCf),'."
At C:\Users\d-mansings\Desktop\Scripted Field Configuration\Script\Prod_UpdateS
cript.ps1:78 char:37
+ $Reader = $Command.ExecuteReader <<<< ()
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
Following is the code I'm using:
Function DatabaseQueries(){
#To connect to the SQL database
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "Server=$IPSource ; Database=$DBNameSource ; User ID=$UserIDSource ; Password=$LoginPwdSource;"
$Connection.Open()
#Query to get the ID of the stored script field from propertyentry
$Command1 = New-Object System.Data.SQLClient.SQLCommand
$Command1.Connection = $Connection
$Command1.CommandText = "SELECT [ID] FROM [dbo].[propertyentry] WHERE [PROPERTY_KEY]='com.onresolve.jira.groovy.groovyrunner:customfields' "
$Reader = $Command1.ExecuteReader()
while ($Reader.Read()) {
$ID = $Reader.GetValue($1)
}
#To get the updated script file
$ScriptDir = $ParentDir + '\Script.txt'
$ScriptData = Get-Content "$ScriptDir"
$Connection.Close()
#Query to update the Script in JIRA database
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
$Command.CommandText = #"
Update [dbo].[propertytext] set [propertyvalue] ='$ScriptData' Where ID=$ID
"#
$Reader = $Command.ExecuteReader()
$Connection.Close()
}

It is difficult to write a complete solution if file contents and database structure are not specified. You surely encountered some kind of SQL injection. SQL Query concatenation is considered harmful and you should avoid it. Use ADO.NET parameters to pass variables ($Command.Parameters.AddWithValue in your example). See the following example:
function Invoke-Sql(
$ConnectionString,
$Query,
$Parameters
) {
$conn = New-Object System.Data.SqlClient.SqlConnection -ArgumentList $ConnectionString
$cmd = New-Object System.Data.SqlClient.SqlCommand -ArgumentList $Query,$conn
$conn.Open()
foreach ($arg in $Parameters.GetEnumerator()){
$cmd.Parameters.AddWithValue($arg.Key, $arg.Value) | Out-Null;
}
$reader = $cmd.ExecuteReader()
if ($reader.Read()) {
[string[]]$columns = 0..($reader.FieldCount-1) |
% { if ($reader.GetName($_)) { $reader.GetName($_) } else { "(no name $_)" } }
do {
$obj = #{}
0..($reader.FieldCount-1) | % { $obj.Add($columns[$_], $reader[$_]) }
New-Object PSObject -Property $obj
} while ($reader.Read())
}
$reader.Dispose()
$cmd.Dispose()
$conn.Dispose()
}
Invoke-Sql `
-ConnectionString "Server=.\SQL2014;Database=Test1;Integrated Security=true" `
-Query 'SELECT Name, Id [ObjectId], Id + 3, #arg FROM IdNameTest' `
-Parameters #{arg = 'Some text'''}
Invoke-Sql `
-ConnectionString "Server=.\SQL2014;Database=Test1;Integrated Security=true" `
-Query 'UPDATE IdNameTest SET Name=#name WHERE Id=#id' `
-Parameters #{name = "'DROP DATABASE Death;! %&#!$"; id=1}

Thanks for the response, I have figured out a way to execute the query by just using a replace function, as it was getting confused between the single inverted commas
select REPLACE(Cast(propertyvalue AS varchar(Max)), '''', '''''') FROM [dbo].[propertytext] WHERE ID=$ID

Related

TVP not being supplied in stored procedure call from Powershell

Can't figure out where I'm going wrong here, the TVP in the stored procedure references #asn but I keep getting the below error/warning, I've tried everything, there is no further details in the sql server logs, any help appreciated.
PowerShell function:
Function Execute-Procedure {
Param(
[Parameter(Mandatory=$true)][array]$p
)
Process {
$connectionString = "Server=;Database=;User ID=;Password=;Trusted_Connection=True"
$conn = New-Object System.Data.SqlClient.SqlConnection $connectionString
$cmd = New-Object System.Data.SqlClient.SqlCommand
$cmd.Connection = $conn
$cmd.CommandTimeout = 0
$pvar = ("V-" + $p.ToString())
$sqlParam = New-Object System.Data.SqlClient.SqlParameter("#asn", $pvar)
$null = $cmd.Parameters.Add($sqlParam, [System.Data.SqlDbType]::NVarChar)
$cmd.CommandText = "EXEC tsp_insert_asn #asn"
try {
$conn.Open()
$cmd.ExecuteNonQuery() | Out-Null
} catch [Exception] {
Write-Warning $_.Exception.Message
} finally {
$conn.Dispose()
$cmd.Dispose()
}
}
}
Stored procedure:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[tsp_insert_asn]
#asn AS [dbo].[tvp_asn] READONLY
AS
BEGIN
INSERT INTO [dbo].[asn]
SELECT asn_id
, name
, size
, location
,GETDATE()
FROM #asn;
END
Error:
WARNING: The parameterized query '(#asn nvarchar(4000))EXEC tsp_insert_asn #asn' expects the parameter '#asn', which was not supplied.
First create a datatable with the same schema as the user-defined type (UDT):
$DataTable = New-Object -TypeName System.Data.DataTable
$DataColumn = New-Object -TypeName System.Data.DataColumn -ArgumentList 'asn_id' , ([String])
$DataTable.Columns.Add($DataColumn)
$DataColumn = New-Object -TypeName System.Data.DataColumn -ArgumentList 'name' , ([String])
$DataTable.Columns.Add($DataColumn)
$DataColumn = New-Object -TypeName System.Data.DataColumn -ArgumentList 'size' , ([String])
$DataTable.Columns.Add($DataColumn)
$DataColumn = New-Object -TypeName System.Data.DataColumn -ArgumentList 'location', ([String])
$DataTable.Columns.Add($DataColumn)
Note that I created ever column as a string. Don't do that! Your columns should match the data type of the UDT columns. At this point I can't remember if you're supposed to use SQL data types or .Net data types, but I'm sure it will be obvious if it doesn't work.
Now populate the data table from your array.
foreach ($i in $p) {
$DataRow = $DataTable.NewRow()
$DataRow.'asn_id' = $i.'asn_id'
$DataRow.'name' = $i.'name'
$DataRow.'size' = $i.'size'
$DataRow.'location' = $i.'location'
$DataTable.Rows.Add($DataRow)
}
Now connect and execution your stored procedure with the datatable as the value of your parameter with the Structured data type.
$conn = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $connectionString
$cmd = New-Object -TypeName System.Data.SqlClient.SqlCommand -ArgumentList $conn
$cmd.CommandType = [System.Data.CommandType]::StoredProcedure
$cmd.CommandText = 'tsp_insert_asn'
$cmd.Parameters.Add("#asn",[System.Data.SqlDbType]::Structured).Value = $DataTable
try {
$conn.Open()
$null = $cmd.ExecuteNonQuery()
}
finally {
$conn.Close()
}

Powershell SQL Select statements using csv variables

In the code below, I'm trying to query a DB with multiple select statements using variables brought in from a csv and load a data-table using a reader.
The code runs without error but does not retrieve any data.
$csv = Import-Csv $filepath
$database = "DBNAME"
$connectionString = "Server=$dataSource;uid=$user; pwd=$pwd;Database=$database;Integrated Security=True;"
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$connection.Open()
$sqlCommand = $connection.CreateCommand()
$Datatable = New-Object System.Data.DataTable
ForEach ($row in $csv){
$query = "Select Emailaddress,Column2 from Users Where [Emailaddress] = '$row.Email'"
$sqlCommand.CommandText = $query
$DataReader = $sqlCommand.ExecuteReader()
$DataTable.Load($DataReader)
}
$DataTable | export-csv "c:\Output\Seereader.csv" -NoTypeInformation
$connection.Close()
This:
$query = "Select Emailaddress,Column2 from Users Where [Emailaddress] = '$row.Email'"
Should probably be this:
$query = "Select Emailaddress,Column2 from Users Where [Emailaddress] = '$($row.Email)'"
Whilst the previous answer works, it is vulnerable to SQL injection.
Obligatory xkcd
If you're not sure what "SQL Injection" is; it's only a very worthy Google away...
i.e. you really need to go and find out!
The correct way...
Parameterise your queries!
# Your query; with a #param
$Query = "SELECT Emailaddress, Column2 FROM [Users] WHERE [Emailaddress] = #emailAddress";
# Set up your basic command
$command = $connection.CreateCommand()
$command.CommandText = $Query
# Fill in the parameters!
$command.Parameters.AddWithValue("#emailAddress", $row.Email)
# Run boy, run!
$results = $command.ExecuteReader()
# Resultification (that's definitely not a made up word)
$table = New-Object System.Data.DataTable
$table.Load($results)
Safe and sound :-)

How do I account for CSV column titles and null rows when trying to import a CSV to SQL Server using PowerShell

I have a large CSV file with 5 columns, the first row being the title of the columns. I'm trying to pass the values to a datatable using powershell that I then pass to a table-value parameter to load the data into my SQL Server instance. My SQL Server table has already been created with the 5 columns, but I am running into issues.
cls
#CSV variables
$csvfile = "C:\Students\test.csv"
$csvdelimiter = ","
$firstRowColumns = $true
#creating the datatable
$dt = New-Object System.Data.Datatable
$reader = New-Object System.IO.StreamReader $csvfile
$columns = (Get-Content $csvfile -First 1).Split($csvdelimiter)
foreach ($column in $columns)
{
if ($firstRowColumns -eq $true)
{
[void]$dt.Columns.Add($column)
$reader.ReadLine()
} else { [void]$dt.Columns.Add() }
}
# Read in the data, line by line
while (($line = $reader.ReadLine()) -ne $null)
{
[void]$dt.Rows.Add($line.Split($csvdelimiter))
}
function ExecSproc
{
param ($Conn, $Sproc, $Parameters=#{})
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandType = [System.Data.CommandType]::StoredProcedure
$SqlCmd.Connection = $Conn
$SqlCmd.CommandText = $Sproc
foreach($p in $Parameters.Keys){
[Void] $SqlCmd.Parameters.AddWithValue("#$p",$Parameters[$p])
}
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter($SqlCmd)
$DataSet = New-Object System.Data.DataSet
[Void] $SqlAdapter.Fill($DataSet)
$SqlConnection.Close()
return $DataSet.Tables[0]
}
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "server='.';database='ActiveStudents';trusted_connection=true;"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
ExecSproc -Conn $SqlConnection -Sproc spInsertStudents -Parameters #{TVP = $dt}
These are the errors being thrown:
Exception calling "Add" with "1" argument(s): "Input array is longer than the number of columns in this table."
At C:\Scripts\ActiveStudentInsert.ps1:24 char:2
Exception calling "Fill" with "1" argument(s): "Conversion failed when converting the nvarchar value '' to data type int.
The data for table-valued parameter "#TVP" doesn't conform to the table type of the parameter. SQL Server error is: 245, state: 1
The statement has been terminated."
Chad Miller made a great script in the Script Gallery that outputs DataTables in a very similar way to the one you are looking to do.
If you used his functions, then it would look something like this:
$dt = Import-Csv $csvfile | Out-DataTable
If you have a really large amount of data, I would recommend you check out the SQL BCP command-line utility.

Pass a variable from Powershell to saved SQL Script

I have a Powershell Function that is being used to run multiple queries in SQL and export as CSVs. Each of these queries relies on a date variable. Is there a way to pass this date variable from Powershell into these SQL Scripts (not stored procedures) using my current setup? Any help is much appreciated!
Function Run-Query
{
param([string[]]$queries,[string[]]$sheetnames)
Begin
{
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database = $Database; User ID = $uid; Password = $pwd;"
Write-host "Connection to database successful."
}#End Begin
Process
{
# Loop through each query
For($i = 0; $i -lt $queries.count; $i++)
{
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
# Use the current index ($i) to get the query
$SqlCmd.CommandText = $queries[$i]
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
# Use the current index ($i) to get the sheetname for the CSV
$DataSet.Tables[0] #| Export-Csv -NoTypeInformation -Path "C:\Users\mbaron\Downloads\$($sheetnames[$i]).csv"
}
}#End Process
End
{
$SqlConnection.Close()
}
}#End function run-query.
You could add a marker in your queries where the data is being used, then do a replace with the relevant date, e.g.:
cls
$date = '1/1/2016'
$query = 'some $$marker$$ script'
$query = $query.replace('$$marker$$', $date )
$query

How do I call a SQL Server stored procedure from PowerShell?

I have a large CSV file and I want to execute a stored procedure for each line.
What is the best way to execute a stored procedure from PowerShell?
This answer was pulled from http://www.databasejournal.com/features/mssql/article.php/3683181
This same example can be used for any adhoc queries. Let us execute the stored procedure “sp_helpdb” as shown below.
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=HOME\SQLEXPRESS;Database=master;Integrated Security=True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = "sp_helpdb"
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$SqlConnection.Close()
$DataSet.Tables[0]
Here is a function that I use (slightly redacted). It allows input and output parameters. I only have uniqueidentifier and varchar types implemented, but any other types are easy to add. If you use parameterized stored procedures (or just parameterized sql...this code is easily adapted to that), this will make your life a lot easier.
To call the function, you need a connection to the SQL server (say $conn),
$res=exec-storedprocedure -storedProcName 'stp_myProc' -parameters #{Param1="Hello";Param2=50} -outparams #{ID="uniqueidentifier"} $conn
retrieve proc output from returned object
$res.data #dataset containing the datatables returned by selects
$res.outputparams.ID #output parameter ID (uniqueidentifier)
The function:
function exec-storedprocedure($storedProcName,
[hashtable] $parameters=#{},
[hashtable] $outparams=#{},
$conn,[switch]$help){
function put-outputparameters($cmd, $outparams){
foreach($outp in $outparams.Keys){
$cmd.Parameters.Add("#$outp", (get-paramtype $outparams[$outp])).Direction=[System.Data.ParameterDirection]::Output
}
}
function get-outputparameters($cmd,$outparams){
foreach($p in $cmd.Parameters){
if ($p.Direction -eq [System.Data.ParameterDirection]::Output){
$outparams[$p.ParameterName.Replace("#","")]=$p.Value
}
}
}
function get-paramtype($typename,[switch]$help){
switch ($typename){
'uniqueidentifier' {[System.Data.SqlDbType]::UniqueIdentifier}
'int' {[System.Data.SqlDbType]::Int}
'xml' {[System.Data.SqlDbType]::Xml}
'nvarchar' {[System.Data.SqlDbType]::NVarchar}
default {[System.Data.SqlDbType]::Varchar}
}
}
if ($help){
$msg = #"
Execute a sql statement. Parameters are allowed.
Input parameters should be a dictionary of parameter names and values.
Output parameters should be a dictionary of parameter names and types.
Return value will usually be a list of datarows.
Usage: exec-query sql [inputparameters] [outputparameters] [conn] [-help]
"#
Write-Host $msg
return
}
$close=($conn.State -eq [System.Data.ConnectionState]'Closed')
if ($close) {
$conn.Open()
}
$cmd=new-object system.Data.SqlClient.SqlCommand($sql,$conn)
$cmd.CommandType=[System.Data.CommandType]'StoredProcedure'
$cmd.CommandText=$storedProcName
foreach($p in $parameters.Keys){
$cmd.Parameters.AddWithValue("#$p",[string]$parameters[$p]).Direction=
[System.Data.ParameterDirection]::Input
}
put-outputparameters $cmd $outparams
$ds=New-Object system.Data.DataSet
$da=New-Object system.Data.SqlClient.SqlDataAdapter($cmd)
[Void]$da.fill($ds)
if ($close) {
$conn.Close()
}
get-outputparameters $cmd $outparams
return #{data=$ds;outputparams=$outparams}
}
Here is a function I use to execute sql commands. You just have to change $sqlCommand.CommandText to the name of your sproc and $SqlCommand.CommandType to CommandType.StoredProcedure.
function execute-Sql{
param($server, $db, $sql )
$sqlConnection = new-object System.Data.SqlClient.SqlConnection
$sqlConnection.ConnectionString = 'server=' + $server + ';integrated security=TRUE;database=' + $db
$sqlConnection.Open()
$sqlCommand = new-object System.Data.SqlClient.SqlCommand
$sqlCommand.CommandTimeout = 120
$sqlCommand.Connection = $sqlConnection
$sqlCommand.CommandText= $sql
$text = $sql.Substring(0, 50)
Write-Progress -Activity "Executing SQL" -Status "Executing SQL => $text..."
Write-Host "Executing SQL => $text..."
$result = $sqlCommand.ExecuteNonQuery()
$sqlConnection.Close()
}
Use sqlcmd instead of osql if it's a 2005 database
Consider calling osql.exe (the command line tool for SQL Server) passing as parameter a text file written for each line with the call to the stored procedure.
SQL Server provides some assemblies that could be of use with the name SMO that have seamless integration with PowerShell. Here is an article on that.
http://www.databasejournal.com/features/mssql/article.php/3696731
There are API methods to execute stored procedures that I think are worth being investigated. Here a startup example:
http://www.eggheadcafe.com/software/aspnet/29974894/smo-running-a-stored-pro.aspx
I include invoke-sqlcmd2.ps1 and write-datatable.ps1 from http://blogs.technet.com/b/heyscriptingguy/archive/2010/11/01/use-powershell-to-collect-server-data-and-write-to-sql.aspx. Calls to run SQL commands take the form: Invoke-sqlcmd2 -ServerInstance "<sql-server>" -Database <DB> -Query "truncate table <table>" An example of writing the contents of DataTable variables to a SQL table looks like: $logs = (get-item SQLSERVER:\sql\<server_path>).ReadErrorLog()
Write-DataTable -ServerInstance "<sql-server>" -Database "<DB>" -TableName "<table>" -Data $logs I find these useful when doing SQL Server database-related PowerShell scripts as the resulting scripts are clean and readable.
Adds CommandType and Parameters to #Santiago Cepas' answer:
function Execute-Stored-Procedure
{
param($server, $db, $spname)
$sqlConnection = new-object System.Data.SqlClient.SqlConnection
$sqlConnection.ConnectionString = 'server=' + $server + ';integrated security=TRUE;database=' + $db
$sqlConnection.Open()
$sqlCommand = new-object System.Data.SqlClient.SqlCommand
$sqlCommand.CommandTimeout = 120
$sqlCommand.Connection = $sqlConnection
$sqlCommand.CommandType= [System.Data.CommandType]::StoredProcedure
# If you have paramters, add them like this:
# $sqlCommand.Parameters.AddWithValue("#paramName", "$param") | Out-Null
$sqlCommand.CommandText= $spname
$text = $spname.Substring(0, 50)
Write-Progress -Activity "Executing Stored Procedure" -Status "Executing SQL => $text..."
Write-Host "Executing Stored Procedure => $text..."
$result = $sqlCommand.ExecuteNonQuery()
$sqlConnection.Close()
}
# Call like this:
Execute-Stored-Procedure -server "enter-server-name-here" -db "enter-db-name-here" -spname "enter-sp-name-here"
I added timeout and show how to reader a scalar or get results using a reader
function exec-query( $storedProcName,$parameters=#{},$conn,$timeout=60){
$cmd=new-object system.Data.SqlClient.SqlCommand
$cmd.CommandType=[System.Data.CommandType]'StoredProcedure'
$cmd.Connection=$conn
$cmd.CommandText=$storedProcName
$cmd.CommandTimeout=$timeout
foreach($p in $parameters.Keys){
[Void] $cmd.Parameters.AddWithValue("#$p",$parameters[$p])
}
#$id=$cmd.ExecuteScalar()
$adapter=New-Object system.Data.SqlClient.SqlDataAdapter($cmd)
$dataset=New-Object system.Data.DataSet
$adapter.fill($dataset) | Out-Null
#$reader = $cmd.ExecuteReader()
#$results = #()
#while ($reader.Read())
#{
# write-host "reached" -ForegroundColor Green
#}
return $dataSet.Tables[0]
}

Resources