Parallel T-SQL execution in PowerShell - sql-server

Can anyone help with this problem? I am referring to a example from the internet for executing T-SQL statements in parallel.
https://www.mssqltips.com/sqlservertip/3539/complete-common-sql-server-database-administration-tasks-in-parallel-with-powershell-v3-workflow/
I want to be able to execute the same T-SQL on the same instance at once for proof of concept work on locking. In order to do this, I have tweaked the script so that I can execute any number of iterations by changing the
while ($counter -le 5)
Here is the full script. Basically the primary statement can be whatever T-SQL you want and this will populate $sqlcmds to have that statement passed through as many iterations as you wish.
Import-Module sqlps -DisableNameChecking;
Set-Location c:
# create a workflow to run multiple sql in parallel
workflow Run-PSQL #PSQL means Parallel SQL {
Param(
[Parameter(Mandatory=$true)]
[string]$ServerInstance,
[Parameter(Mandatory=$false)]
[string]$Database,
[Parameter(Mandatory=$true)]
[string[]]$Query # a string array to hold t-sqls
)
foreach -parallel ($q in $query) {
Invoke-Sqlcmd -ServerInstance $ServerInstance -Database $Database -Query $q -QueryTimeout 60000;
}
} # Run-PSQL
# prepare a bunch of sql commands in a string arrary
#####new bit to make it dynamic sql multiple times
[string[]]$sqlcmds
$sqlcmds = ""
$counter = 0
do {
"Starting Loop $Counter"
$PrimaryStatement = '"SELECT TOP 1 * FROM sys.objects"'
if ($counter -eq 5) {
$sqlcmds = $sqlcmds + "$PrimaryStatement"
Write-Host "this is what sqlcmds is $sqlcmds loop 5"
} else {
$sqlcmds = $sqlcmds + "$PrimaryStatement,``"
Write-Host "this is what sqlcmds is now $sqlcmds"
}
$counter++
} while ($counter -le 5)
# now we can run the workflow and measure its execution duration
$dt_start = Get-Date; #start time
Run-PSQL -Server &&&&&&& -Database master -Query $sqlcmds;
$dt_end = Get-Date; #end time
$dt_end - $dt_start; # find execution duration
When this is executed, I get this message:
Run-PSQL : Cannot bind argument to parameter 'Query' because it is an empty string.

There are a few minor corrections I had to make and below is the final code that seems to work as expected
Moved the first { since it got commented!
Removed $sqlcmds = ""
Changed how SQL is aggregated in the array $sqlcmds
Removed the if/else inside as it did not seem to serve a
purpose
Changed the text printed
Import-Module sqlps -DisableNameChecking;
Set-Location c:
# create a workflow to run multiple sql in parallel
workflow Run-PSQL #PSQL means Parallel SQL
{
Param(
[Parameter(Mandatory=$true)]
[string]$ServerInstance,
[Parameter(Mandatory=$false)]
[string]$Database,
[Parameter(Mandatory=$true)]
[string[]]$Query #a string array to hold t-sqls
)
foreach -parallel ($q in $query) {
Invoke-Sqlcmd -ServerInstance $ServerInstance -Database $Database -Query $q -QueryTimeout 60000;
}
} # Run-PSQL
# prepare a bunch of sql commands in a string arrary
##### new bit to make it dynamic sql multiple times
[string[]]$sqlcmds = #()
$counter = 0
do {
"Starting Loop $Counter"
$PrimaryStatement = 'SELECT TOP 1 * FROM sys.objects'
$sqlcmds += "$PrimaryStatement"
Write-Host ("this is what sqlcmds has [$($sqlcmds.Count)] statements at loop counter [$Counter]")
$counter++
} while ($counter -le 5)
# now we can run the workflow and measure its execution duration
$dt_start = Get-Date; #start time
Run-PSQL -Server 'myserver\myinstance' -Database master -Query $sqlcmds;
$dt_end = Get-Date; #end time
$dt_end - $dt_start; # find execution duration

Related

Query on ##IDLE or ##IO_BUSY fails on some databases from Invoke-Sqlcmd

Running the same query on multiple database sometimes fails when ##IDLE or ##IO_BUSY is queried. It works on 6 databases and fails on 4 databases. When using -Verbose, VERBOSE: Arithmetic overflow occurred. is output, but there is no indication of what object overflowed.
PS C:\src\Modules> $Query
SELECT
SERVERPROPERTY('ServerName') AS ServerName
,SERVERPROPERTY('ComputerNamePhysicalNetBIOS') AS ComputerNamePhysicalNetBIOS
,##IDENTITY AS I_DENTITY
,##IDLE AS I_DLE
,##IO_BUSY AS I_O_BUSY
,##MAX_PRECISION AS MAX_PRECISION
PS C:\src\Modules> $ServerInstance = 'DBSERVER1'
PS C:\src\Modules> Invoke-Sqlcmd -Query $Query -ServerInstance $ServerInstance -Verbose
ServerName : DBSESRVER1
ComputerNamePhysicalNetBIOS : DGEDW284
I_DENTITY :
I_DLE : -869467476
I_O_BUSY : 1767922
MAX_PRECISION : 38
PS C:\src\Modules> $ServerInstance = 'DBSERVER2'
PS C:\src\Modules> Invoke-Sqlcmd -Query $Query -ServerInstance $ServerInstance -Verbose
PS C:\src\Modules>
The code used sets $Results to $null on the failing databases.
try {
$Results = Invoke-Sqlcmd -Query $Query `
-ServerInstance $Instance `
-ErrorAction SilentlyContinue
if ($null -ne $Results) {
foreach ($Result in $Results) {
$result | Add-Member -NotePropertyName instance -NotePropertyValue $Instance
$ResultList += $result
}
} else {
Write-Verbose "Results is set to `$null for instance $Instance"
}
}
catch {
Write-Verbose "in catch"
}
Known bug on ##IO_BUSY
Note copied from docs for posterity- If the time returned in ##CPU_BUSY, or ##IO_BUSY exceeds approximately 49 days of cumulative CPU time, you receive an arithmetic overflow warning. In that case, the value of ##CPU_BUSY, ##IO_BUSY and ##IDLE variables are not accurate.

Invoke-SqlCmd generates an exception when used in a loop that contains another Invoke-SqlCmd

I'm having an issue with calling Invoke-SqlCmd when it contains a second Invoke-SqlCmd call:
function Get-Foo
{
$query=`
#"
WITH data AS
(
SELECT 1 ID, 'A' NAME
UNION ALL
SELECT 2 ID, 'B' NAME
UNION ALL
SELECT 3 ID, 'C' NAME
)
SELECT *
FROM data
"#
Invoke-SqlCmd -ServerInstance "SERVER" -Database "DATABASE" -Query $query
}
function Get-Bar
{
param
(
[int]$ParentId
)
$query=`
#"
WITH data AS
(
SELECT 1 ID, 'A' NAME, 1 PARENT_ID
UNION ALL
SELECT 2 ID, 'B' NAME, 1 PARENT_ID
UNION ALL
SELECT 3 ID, 'C' NAME, 2 PARENT_ID
)
SELECT *
FROM data
WHERE parent_id = $ParentId
"#
Invoke-SqlCmd -ServerInstance "SERVER" -Database "DATABASE" -Query $query
}
Get-Foo | ForEach-Object {
Get-Bar -ParentId $_.ID
}
The first iteration of the outer loop works fine, but when it attempts the the second iteration, and exception is generated:
Invoke-SqlCmd : The WriteObject and WriteError methods cannot be
called from outside the overrides of the BeginProcessing,
ProcessRecord, and EndProcessing methods, and they can only be called
from within the same thread. Validate that the cmdlet makes these
calls correctly, or contact Microsoft Customer Support Services. At
Untitled-1.ps1:18 char:3
+ Invoke-SqlCmd -ServerInstance "SERVER" -Database "DATABASE ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidResult: (SERVER:PSObject) [Invoke-Sqlcmd], PSInvalidOperationExceptio n
+ FullyQualifiedErrorId : ExecutionFailed,Microsoft.SqlServer.Management.PowerShell.GetScriptCommand
This syntax works, however:
$foo = Get-Foo
$foo | ForEach-Object {
Get-Bar
}
I'm guessing that I need to close the first Invoke-SqlCmd, but perhaps there is another solution.
This is related to the way the PowerShell pipeline works (see Understanding pipelines), and a limitation in Invoke-SqlCmd where it doesn't like running more than one query in parallel.
To illustrate, let's rewrite your example as follows:
function Invoke-SqlCmd
{
write-output "aaa";
write-output "bbb";
write-output "ccc";
}
function Get-Foo
{
write-host "Get-Foo started";
Invoke-SqlCmd;
write-host "Get-Foo finished";
}
function Get-Bar
{
param( $value )
write-host "Get-Bar '$value'";
}
and now run this script:
write-host "using pipelines"
Get-Foo | foreach-object { Get-Bar $_ }
which gives this output:
using pipelines
Get-Foo started
Get-Bar 'aaa'
Get-Bar 'bbb'
Get-Bar 'ccc'
Get-Foo finished
If you look at the output, you can see that the "Get-Bar" commands are being called while Get-Foo is still "active" in the pipeline.
Compare to this:
write-host "using immediate evaluation"
(Get-Foo) | foreach { Get-Bar $_ }
which gives:
using immediate evaluation
Get-Foo started
Get-Foo finished
Get-Bar 'aaa'
Get-Bar 'bbb'
Get-Bar 'ccc'
where Get-Foo is executed to completion before the values are piped into Get-Bar.
Your specific error is due to your script using multiple concurrent pipeline steps but Invoke-SqlCmd not supporting multiple concurrent pipeline steps. Fortunately you can tell PowerShell to use "immediate evaluation" instead in a few different ways, including the approaches mentioned in other answers:
assigning the expression to a temporary variable first:
$foo = Get-Foo;
$foo | ForEach-Object { Get-Bar $_ }
using the SubExpression operator in your script (note the $ is optional in some cases as per Is "Dollar-sign" optional in powershell "$()"?)
(Get-Foo) | Foreach-Object { Get-Bar $_ }
or wrap Invoke-SqlCmd in a subexpression inside your function. (This one is a bit misleading because Get-Foo is still active in the pipeline but Invoke-SqlCmd is evaluated immediately).
function Get-Foo
{
write-host "Get-Foo started";
(Invoke-SqlCmd);
write-host "Get-Foo finished";
}
I cannot explain why this isn't working (although it looks like the threads are being mixed and the .NET provider cannot handle it properly).
What does work (but this is basically the same as your proposal of assigning it to a variable first) is to surround Get-Foo by parentheses:
(Get-Foo) | Foreach-Object { Get-Bar }
You can also experiment with the -Begin, -Process and -End parameters of Foreach-Object:
Get-Foo | ForEach-Object -Process {$_} -End { Get-Bar }
Based on Defending Invoke-SqlCmd, I decided to implement my own Invoke-SqlCmd cmdlet:
function Invoke-SqlCmd
{
[cmdletbinding()]
param
(
[string]$ServerInstance,
[string]$Database,
[string]$Query
)
Write-Debug $MyInvocation.MyCommand.Name
Write-Debug "$ServerInstance.$Database"
try {
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server=$ServerInstance;database=$Database;trusted_connection=true;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
$Command.CommandText = $Query
$DataAdapter = New-Object System.Data.SqlClient.SqlDataAdapter $Command
$Dataset = New-Object System.Data.Dataset
$DataAdapter.Fill($Dataset) | Out-Null # suppress record count
$Dataset.Tables[0]
}
catch
{
Write-Error $_
}
finally
{
$Connection.Close()
}
}
This implementation will be used instead of the 'regular' Invoke-SqlCmd cmdlet in my functions.
The loop works as expected.

Powershell SQL Merge on output?

I have scrubbed together some powershell to scan a directory list the contents and then insert to a MS SQL (2017) table without creating duplicates.
I am unable to to amend the MERGE to delete old/orphaned row in sql based on a compare of the current directory:
Add-Type -AssemblyName "System.Web"
function Get-FriendlySize {
param($Bytes)
$sizes='Bytes,KB,MB,GB,TB,PB,EB,ZB' -split ','
for($i=0; ($Bytes -ge 1kb) -and
($i -lt $sizes.Count); $i++) {$Bytes/=1kb}
$N=2; if($i -eq 0) {$N=0}
"{0:N$($N)} {1}" -f $Bytes, $sizes[$i]
}
#Declare Servername
$sqlServer='localhost'
#Invoke-sqlcmd Connection string parameters
$params = #{'server'='localhost';'Database'='kentico'}
#Fucntion to manipulate the data
Function writeDiskInfo
{
param ($FileName, $FileTitle, $FileDescription,$FileExtension,$FileMimeType,$FilePath,$FileSize,$FileImageWidth,$FileImageHeight,$FileGUID,$FileLibraryID,$FileSiteID,$FileCreatedByUserID,$FileCreatedWhen,$FileModifiedByUserID, $FileModifiedWhen,$FileCustomData,$FileDate)
# Data preparation for loading data into SQL table
$InsertResults = #"
Merge dbo.Media_File AS T
USING (SELECT FileName = '$FileName') AS S
ON S.FileName = T.FileName
WHEN NOT MATCHED BY TARGET THEN
INSERT (FileName,FileTitle,FileDescription,FileExtension,FileMimeType,FilePath,FileSize,FileImageWidth,FileImageHeight,FileGUID,FileLibraryID,FileSiteID,FileCreatedByUserID,FileCreatedWhen,FileModifiedByUserID,FileModifiedWhen,FileCustomData,FileDate)
VALUES ('$FileName','$FileTitle','$FileDescription','$FileExtension','$FileMimeType','$FilePath','$FileSize','$FileImageWidth','$FileImageHeight', '$FileGUID','$FileLibraryID','$FileSiteID','$FileCreatedByUserID','$FileCreatedWhen','$FileModifiedByUserID', '$FileModifiedWhen','$FileCustomData',CAST('$FileDate' AS datetime))
**This is where I get stuck**
WHEN NOT MATCHED BY SOURCE AND
(SELECT FileName = '$FileName') = T.FileName THEN
DELETE;
"#
#call the invoke-sqlcmdlet to execute the quer'
Invoke-sqlcmd #params -Query $InsertResults
}
#Query WMI query to store the result in a varaible
$dp = (get-childitem -Path C:\inetpub\wwwroot\Kentico11\CMS\t\media\test\ -Recurse -file |Select-Object -property #{Label='FileName';Expression={$_.baseName}},#{Label='FileTitle';Expression={$_.baseName}},#{N='FileDescription';E={"bob"}},#{Label='FileExtension';Expression={$_.Extension }},#{N='FileMimeType';E={[System.Web.MimeMapping]::GetMimeMapping("C:\inetpub\wwwroot\Kentico11\CMS\Enert\media\test\Compliance Analysis 2018-13-04.pdf")}},#{Label='FilePath';Expression= {($_.fullname.remove( 0, 58).replace( '\' ,'/'))}},#{N='FileSize';E={'1234'}},#{N='FileImageWidth';E={(Null)}},#{N='FileImageHeight';E={(Null)}},#{N='FileGUID';E={[guid]::newguid()}},#{N='FileLibraryID';E={("15")}},#{N='FileSiteID';E={("3")}},#{N='FileCreatedByUserID';E={("53")}},#{Label='FileCreatedWhen';Expression={$_.creationtime}},#{N='FileModifiedByUserID';E={("53")}}, #{Label='FileModifiedWhen';Expression={$_.creationtime}},#{N='FileCustomData';E={(Null)}},#{Label='FileDate';Expression={($_.baseName.Substring($_.basename.Length -13, 13))}})
#Loop through array
foreach ($item in $dp)
{
#Call the function to transform the data and prepare the data for insertion
writeDiskInfo $item.FileName $item.FileTitle $item.FileDescription $item.FileExtension $item.FileMimeType $item.FilePath $item.FileSize $item.FileImageWidth $item.FileImageHeight $item.FileGUID $item.FileLibraryID $item.FileSiteID $item.FileCreatedByUserID $item.FileCreatedWhen $item.FileModifiedByUserID $item.FileModifiedWhen $item.FileCustomData $item.FileDate
}
Invoke-Sqlcmd #params -Query "SELECT * FROM Media_File" | format-table -AutoSize

how to write powershell script to list the available databases from multiple servers?

Requirement : My requirement is I have to list the available databases from 150 servers. Each server has minimum 1 and maximum 15 instances.
Below script is working only for instances listed in sqlserver.txt but I need to fetch multiple instances across multiple servers.
Help is highly appriciated.
ForEach ($instance in Get-Content "C:\PowerSQL\SQL_Servers.txt")
{
Import-Module SQLPS -DisableNameChecking
Invoke-SQLcmd -Server $instance -Database master 'select ##servername as InstanceName,name as DatabaseName,state_desc as DBStatus from sys.databases' | Format-Table
}
You can use this script to find all reachable instances on your network and running your query there:
Import-Module SQLPS -DisableNameChecking
$servers = [System.Data.Sql.SqlDataSourceEnumerator]::Instance.GetDataSources()
ForEach ($i in $servers) {
$instance = $i.ServerName+"\"+$i.InstanceName
Invoke-SQLcmd -Server $instance -Database master 'select ##servername as InstanceName,name as DatabaseName,state_desc as DBStatus from sys.databases' | Format-Table
}
If you need only server name to pass then use $instance = $i.ServerName. Part of code was taken from here long time ago.
EDIT
With writing in CSV file and error catching:
Import-Module SQLPS -DisableNameChecking
$servers = [System.Data.Sql.SqlDataSourceEnumerator]::Instance.GetDataSources()
$results = #()
ForEach ($i in $servers) {
$instance = $i.ServerName+"\"+$i.InstanceName
try {
$sqlres = Invoke-SQLcmd -Server $instance -Database master 'select ##servername as InstanceName,name as DatabaseName,state_desc as DBStatus from sys.databases'
ForEach($st in $sqlres) {
$instanceinfo = #{
InstanceName = $st.InstanceName
DatabaseName = $st.DatabaseName
DBStatus = $st.DBStatus
}
$results += New-Object PSObject -Property $instanceinfo
}
} catch {
"error when running Invoke-SQLcmd "+$instance
Write-Host($error)
}
}
$results | export-csv -Path D:\sql_instances_info.csv -NoTypeInformation
Im not sure what is the problem here. You can put all servers/instances in txt file and iterate:
#array of addresses, this can be fetched from file
$list = "localhost\SQL2014",".\SQL2014","(local)\SQL2014" #MyServer\MyInstance
$list | `
% { Invoke-Sqlcmd -Server $_ -Database master 'select ##servername as InstanceName,name as DatabaseName,state_desc as DBStatus from sys.databases' } | `
Format-Table -AutoSize
If those are remote servers without integrated security you would need to pass -UserName and -Password arguments.

Powershell DBCC CheckDB from Powershell

I was wondering if anyone could help with the following code shown. I'm basically trying to get this code re-hashed if possible to allow me to run it against a set of server names supplied in a text file named "servers.txt".
The DBCC should be run by the PS script and run against all DB for that servername. I'm not up to speed enough with PS to understand how to do this for this script.
How to change it allow to plug in values instead of being hardcoded for each servername?
I've read a bit around this and looked at the Invoke-Sql command which I believe is a SQL 2008 extension to PS.
Unfortunately the PS environment is from a SQL 2005 box and I dont have the power to get this moved so dont think ill be able to use invoke
Please see the original code and then my experiment to try and get it to run using invoke.
$ScriptName = $myInvocation.MyCommand.Name
[void][reflection.assembly]::LoadWithPartialName("System.Data.SqlClient")
$ConnString = "Server=DB-OCC05;Integrated Security=SSPI;Database=master;Application Name=$ScriptName"
$MasterConn = new-object ('System.Data.SqlClient.SqlConnection') $ConnString
$MasterCmd = new-object System.Data.SqlClient.SqlCommand
$MasterCmd.Connection = $MasterConn
$SqlDBCC = "DBCC CHECKDB(master) WITH TABLERESULTS"
$MasterCmd.CommandText = $SqlDBCC
$MasterConn.Open()
$Rset = $MasterCmd.ExecuteReader()
If ($Rset.HasRows -eq $true) {
While ($Rset.Read()) {
$line = $Rset["MessageText"]
If ($Rset["Level"] -gt 10) {
Write-Host $line -backgroundcolor Yellow -foregroundcolor Red
} else {
Write-Host $line
}
}
$Rset.Close()
}
$MasterConn.Close()
And then my test running from SQL 2005 environment:
Invoke-Sqlcmd -Query "SELECT GETDATE() AS TimeOfQuery;" -ServerInstance "MyComputer\MyInstance"
And also tried this test:
gc "C:\Powershell\Servers.txt" | foreach-object {Invoke-Sqlcmd "DBCC checkdb;" -ServerInstance "$_\MyInstance"}
But the above test runs didnt work cause of the:
The term 'Invoke-Sqlcmd' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path.
A few modifications to your script. Everything is basically the same except for the connection string and the few lines at the bottom for loading your servers.txt file (a text file with one line per instance) and enumerating its content:
function Execute-DBCC
{
param (
[parameter(Mandatory = $true)][string]$serverInstance
)
$connString = "Server=$serverInstance;Integrated Security=SSPI;Database=master;Application Name=$ScriptName"
$masterConn = new-object ('System.Data.SqlClient.SqlConnection') $connString
$masterCmd = new-object System.Data.SqlClient.SqlCommand
$masterCmd.Connection = $masterConn
$masterCmd.CommandText = "EXECUTE master.sys.sp_MSforeachdb 'DBCC CHECKDB([?]) WITH TABLERESULTS'"
$masterConn.Open()
$reader = $masterCmd.ExecuteReader()
if ($reader.HasRows -eq $true)
{
while ($reader.Read())
{
$messageText = $reader["MessageText"]
if ($reader["Level"] -gt 10)
{ Write-Host $messageText -backgroundcolor Yellow -foregroundcolor Red }
else
{ Write-Host $messageText }
}
$reader.Close()
}
$masterConn.Close()
}
[void][reflection.assembly]::LoadWithPartialName("System.Data.SqlClient")
$servers = #(Get-Content ".\servers.txt")
$servers | %{
Execute-DBCC $_
}

Resources