Return SQL Query as Array in Powershell - sql-server

I have a SQL 2008 Ent server with the databases "DBOne", "DBTwo", "DBThree" on the server DEVSQLSRV.
Here is my Powershell script:
$DBNameList = (Invoke-SQLCmd -query "select Name from sysdatabases" -Server DEVSQLSRV)
This produces my desired list of database names as:
Name
-----
DBOne
DBTwo
DBThree
I has been my assumption that anything that is returned as a list is an Array in Powershell. However, when I then try this in Powershell:
$DBNameList -contains 'DBTwo'
It comes back has "False" instead of "True" which is leading me to believe that my list is not an actual array.
Any idea what I'm missing here?
Thanks so much!
Emo

I'd do this:
$DBNameList = #(Invoke-SQLCmd -query "select Name from sysdatabases" -Server DEVSQLSRV) | select-object -expand Name
That will give you an array of names.The option -contains should work fine.

What's missing from the original post is some type of conversion, from an object to an array.
Powershell outputs the result of $DBNameList because it kinda interprets the object. But if you need to manipulate this object and identify a specific item from it, this is the method I use:
$Itm = "DBTwo"
$DBNameList = #(Invoke-SQLCmd -query "select Name from sysdatabases" -Server DEVSQLSRV)
$NameList = #($DBNameList | select-object -ExpandProperty Name)
$Name = ($NameList.Split()).Contains($Itm)
Write-Output $Name
True
I have been looking for this myself for a while and finally worked it out, so I hope it helps someone else!

The Name header suggests it's a single object with a Name property which is an array.
I reckon initialize an empty PS array :
$DBNameList = (Invoke-SQLCmd -query "select Name from sysdatabases" -Server DEVSQLSRV)
[Array]$DbNames = #()
$DBNameList.Name | ForEach-Object {$DbNames += $_}
$DbNames -contains "DBTwo"
Any luck?

Your code...
$DBNameList = (Invoke-SQLCmd -query "select Name from sysdatabases" -Server DEVSQLSRV)
...gives you Datarows back..
You can check this with
$DBNameList | Get-Member
You can also see that there is a property wit the name "Name".
If you want to check if one of the datarows in your $DBNameList contains the Name of 'DBTwo' you would need to write it the following:
$DBNameList.Name -contains 'DBTwo'

In case anybody else ended up here because they were terrified they were going to have to hand-type the name of every property in the DataRow object response in order to get several columns into an array, have no fear, there is a handy property called "ItemArray" that provides what you need.
(Invoke-SQLCmd -query "select Name from sysdatabases").ItemArray -contains 'DBTwo'
True
There are plenty of great answers here that solve this particular OP's quandry, but when the column list get's long, this makes things a lot simpler.
(Invoke-SQLCmd -query "select DBID,Name,Version from sysdatabases")[0].ItemArray -join ','
1,master,852

Reading all these complicated answers and I realized there is a simple one:
$DBNameList.Name -contains 'DBTwo'
This should return true.
Your initial pull of the data (the database names) returns an array of objects. Each object has many properties. But in your logic test, you are trying to compare the entire object to a single string. You need to compare a single property of the object (.Name) to the string.

Still very new to Powershell (less than two weeks): I suggest that you try this, if your query contains multiple columns and rows... Multi-Dimensional Arrays. This was my first attempt at this, and after checking the web, given that I could not find a simple straight forward solution, I ended up writing my own solution. Here's the full set of sample code for you to experiment with and use.
Full set of sample code below....
#############################################################################################
# RDSago
# RDSago#gmail.com
# 09/20/2014
#############################################################################################
#
# Capturing database size information from a collection of servers
# and returning that back to an array that can be used to populate
# a SQL table that can be used for monitoring database growth remotely.
# RDSago, RDSago#gmail.com
#
# Note, SQL data retrieved in this manner, does not have to be parsed
# before it is consumed and used elsewhere, just like any array you have defined.
# The data only needs to be addressed by its ".identityname" captured in the
# array $queryResults (shown below).
#
############################################################################################
#############################################################################################
# T-SQL for creating table to hold data returned
#
# CREATE TABLE [dba].[tbl_dbfilesize](
# [ServerNameInstance] [varchar](20) NULL,
# [DatabaseName] [varchar](30) NULL,
# [DataFileSizeMB] [numeric](20, 0) NULL,
# [LogFileSizeMB] [numeric](20, 0) NULL,
# [TotalDatabaseSizeMB] [numeric](20, 0) NULL,
# [CollectionDate] [date] NULL
# ) ON [PRIMARY]
#############################################################################################
Try
{
#define your connection points
# first create an array that will hold the server/instance name of the servers you wish to audit
# the first sever assumes a named instance, the second a default instance name.
$SourceServerName = #("ServerName01/InstanceName", "ServerName02", "ServerName03") # Server you will retrieve data from
#next define the server connection for where you will write your data back to
$TargetServerInstance = "TaretServerName"
# define your sql query that will be used to pull data from SQL on the Source Server
$qryDatabaseInfo = "
SELECT ##ServerName as ServerNameInstance,
DB.name as DatabaseName,
SUM(CASE WHEN type = 0 THEN MF.size * 8 / 1024 ELSE 0 END) AS DataFileSizeMB,
SUM(CASE WHEN type = 1 THEN MF.size * 8 / 1024 ELSE 0 END) AS LogFileSizeMB,
SUM(CASE WHEN type = 1 THEN MF.size * 8 / 1024 ELSE 0 END) + SUM(CASE WHEN type = 0 THEN MF.size * 8 / 1024 ELSE 0 END) AS TotalDatabaseSizeMB
FROM sys.master_files MF
JOIN sys.databases DB ON DB.database_id = MF.database_id
GROUP BY DB.name
ORDER BY DB.NAME ASC
"
#Loop through all the servers you wish to audit
ForEach ($SourceServerName in $SourceServerNames)
{
#execute query to pull data from server into an array
$queryResults = #(Invoke-SQLCmd -query $qryDatabaseInfo -Server $SourceServerInstance)
# Next, construct your insert statement from data in your $queryresults array.
Foreach ($queryResult in $queryResults)
{
$query = "
Insert Into [DBS_AUDIT_SERVERS].[dba].[tbl_dbfilesize]
([ServerNameInstance],
[DatabaseName],
[DataFileSizeMB],
[LogFileSizeMB],
[TotalDatabaseSizeMB],
[CollectionDate])
Values
(" +
"'" + $SourceServerInstance + "'," +
"'" + $queryResult.DatabaseName + "'," +
"'" + $queryResult.DataFileSizeMB + "'," +
"'" + $queryResult.LogFileSizeMB + "'," +
"'" + $queryResult.TotalDatabaseSizeMB + "'," +
"'" + $Date + "'" +
")"
""
#execute insert statement for sql
Invoke-Sqlcmd -Query $query -ServerInstance $TargetServerInstance
}
}
}
Catch [Exception]
{
$ErrorMessage = $_.Exception.Message
Write-Host $ErrorMessage
}
Finally
{
Write-Host "Completed Successfully"
}
Return 0;

Related

Why is Powershell Import Data placing an empty row in SQL Server?

I'm using a Powershell script to import API data into SQL Server. At the beginning of the script, the table within the database is truncated--
$TruncateTable = #"
TRUNCATE TABLE [dbo].[Table1]
"#
Invoke-sqlcmd #params -Query $TruncateTable
Then, the data is imported into SQL Server using a foreach loop with a nested if-else statement and some regex to isolate needed data.
foreach ($r in $roles) {
$InsertRoleResults = #"
INSERT INTO [dbo].[Table1]([roleID],[roleLocation],[roleName])
VALUES ('$roleId','$rolelocation','$rolename')
"#
# If role value contains brackets: assign variable to data within brackets/ variable to data after brackets
if($r.name -match '\[')
{
[int]$roleId = $r.id
$rolelocation = $r.name -replace '(\[)(.+)(\])(.+)', '$2'
$rolename = $r.name -replace '\[.+\]\ ', ''
Invoke-sqlcmd #params -Query $InsertRoleResults
}
else
# if data does not contain brackets, assign Null to rolelocation, entire string to rolename
{
[int]$roleId = $r.id
$rolelocation = $null
$rolename = $r.name
Invoke-sqlcmd #params -Query $InsertRoleResults
}
}
In both blocks, I also take the role id number ($r.id) and convert to INT and then reassign to $roleID.
Then it inserts into SQL Server. I have this problem where I keep getting this happening:
If I Write-Host the output of the id values, it starts with 1, not zero. So my question is, where is this zero value coming from?

Run a sql query from PS, and loop in criteria in the where clause

I have code where I'm creating an index file by parsing out pieces of a file name.
This file name:
25643245_AjaWar_Prav_2_FT_20200701.pdf
Will create this line in the index file:
256432245|ST|W-HIGH SCHOOL TRANSCRIPT|##TEST-LOCATION\25643245_AjaWar_Prav_2_FT_20200701.pdf
The problem is that the first parse of '256432245' isn't a primary key in our database, so I have to convert to a primary key, then store the primary key into the index file in place of '256432245'
I have part of the query that builds the index file working correctly, but not the part that queries and returns the converted ID. If I run only the part of the query that returns just one ID, that also works. I'm having a problem getting the query to work within the "foreach".
I currently get this a result:
|ST|W-HIGH SCHOOL TRANSCRIPT|##TEST-LOCATION\25643245_AjaWar_Prav_2_FT_20200701.pdf
When I want to get:
8992004|ST|W-HIGH SCHOOL TRANSCRIPT|##TEST-LOCATION\25643245_AjaWar_Prav_2_FT_20200701.pdf
Where '8992004' is the SPRIDEN_ID is the result of the sql query.
Thanks for any help you can provide.
foreach ($Filename in Get-ChildItem $ImagePath)
{
$Arr = $Filename -split '_'
$reworkedfilename = $Arr[0] + '_' + $Arr[1] + '_' + $Arr[2] + '_' + $Arr[3] + '_' + $Arr[4] + '_' + $Arr[5]
##$reworkedarray2 = $Arr[0] -replace ".pdf", "";
Write-host $Arr[0] ##this works because I can see the non-primary ID being returned
#Find Each SPRIDEN_ID translated from CAID
add-type -AssemblyName System.Data.OracleClient
$username = "U"
$password = "P"
$data_source = "DS"
$connection_string = "User Id=$username;Password=$password;Data Source=$data_source"
$statement = "
Select Distinct SPRIDEN_ID
from SARACMT, SPRIDEN
where
SPRIDEN_PIDM = SARACMT_PIDM
and SPRIDEN_CHANGE_IND is null
AND SARACMT_COMMENT_TEXT = '$Arr[0]'
"
##The "AND SARACMT_COMMENT_TEXT = '$Arr[0]'" doesn't work because nothing is being returned in the index file
try{
$con = New-Object System.Data.OracleClient.OracleConnection($connection_string)
$con.Open()
$cmd = $con.CreateCommand()
$cmd.CommandText = $statement
$result = $cmd.ExecuteReader()
# Do something with the results...
$ArrConverted = while ($result.Read()) {
$result.GetString(0)
}
} catch {
Write-Error (“Database Exception: {0}`n{1}” -f `
$con.ConnectionString, $_.Exception.ToString())
} finally{
if ($con.State -eq ‘Open’) { $con.close() }
}
$outputline = $ArrConverted + '|' + $Arr[4] + '|' + $DocType + '|##'+ $ImagePath + $reworkedfilename | out-file -filepath $IndexFilePath -Encoding "ascii" -append
#>
}
Your issue is to do with how you’re trying to inject variable values into your sql query string:
$statement = "
Select Distinct SPRIDEN_ID
from SARACMT, SPRIDEN
where
SPRIDEN_PIDM = SARACMT_PIDM
and SPRIDEN_CHANGE_IND is null
AND SARACMT_COMMENT_TEXT = '$Arr[0]'
"
If you add a write-host $statement after this line you’ll see it’s replacing just the $Arr part and not the $Arr[0] part, so your query contains something like:
Select Distinct SPRIDEN_ID
from SARACMT, SPRIDEN
where
SPRIDEN_PIDM = SARACMT_PIDM
and SPRIDEN_CHANGE_IND is null
AND SARACMT_COMMENT_TEXT = '25643245 AjaWar Prav 2 FT 20200701[0]'
instead of:
Select Distinct SPRIDEN_ID
from SARACMT, SPRIDEN
where
SPRIDEN_PIDM = SARACMT_PIDM
and SPRIDEN_CHANGE_IND is null
AND SARACMT_COMMENT_TEXT = '25643245’
To get it to replace the value of $Arr[0] you can surround it with $( ... ) to use “command substitution” (see https://learn.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-string-substitutions?view=powershell-7#command-substitution) so your query becomes:
$statement = "
Select Distinct SPRIDEN_ID
from SARACMT, SPRIDEN
where
SPRIDEN_PIDM = SARACMT_PIDM
and SPRIDEN_CHANGE_IND is null
AND SARACMT_COMMENT_TEXT = '$($Arr[0])’
Having said that, you’d be much better off using a parameterised query rather than building a dynamic sql string (see Oracle Parameterized query in c#) because as your code stands it’s vulnerable to a sql injection attack if you come across a deliberately mischievous file name (e.g. ‘ or 1='1_AjaWar_Prav_2_FT_20200701.pdf).
See https://blogs.oracle.com/sql/what-is-sql-injection-and-how-to-stop-it for more about sql injection attacks.
Just by sheer luck I figured it out.
I created a variable for $Arr[0]
$Arr0 = $Arr[0]
Then put the new variable in the where clause of where clause of the sql statement:
AND SARACMT_COMMENT_TEXT = '$Arr0'
This ran the query on each item parsed out during the foreach.

Load data into memory or select multiple times

I have a process that runs every hour, as a part of the process it iterating on a text file that contains about 100K strings and it need to check if each line already exists in specific table in a SQL Server database that has about 30M records.
I have 2 options:
Option 1: SELECT all strings from my table and load it into memory and then during the process it will check for each line in the file if it exists in the data.
Downside: It eats up the machine memory.
Option 2: check if each line in the 100K text file is found in the database (assumes table is indexed correctly).
Downside: It will require multiple requests (100K requests) to database.
Questions:
If I'm using option 2, can SQL Server handle this number of requests?
What is the preferred way in order to overcome this issue?
Below is PowerShell example code for another option; bulk insert the strings into temp table and perform the lookups as a single set-based SELECT query. I would expect this method to typically run a few seconds, depending on your infrastructure.
$connectionString = "Data Source=.;Initial Catalog=YourDatabase;Integrated Security=SSPI"
$connection = New-Object System.Data.SqlClient.SqlConnection($connectionString)
# load strings from file into a DataTable
$timer = [System.Diagnostics.Stopwatch]::StartNew()
$dataTable = New-Object System.Data.DataTable
($dataTable.Columns.Add("StringData", [System.Type]::GetType("System.String"))).MaxLength = 20
$streamReader = New-Object System.IO.StreamReader("C:\temp\temp_strings.txt")
while ($streamReader.Peek() -ge 0) {
$string = $streamReader.ReadLine()
$row = $dataTable.NewRow()
[void]$dataTable.Rows.Add($row)
$row[0] = $string
}
$streamReader.Close()
Write-Host "DataTable load completed. Duration $($timer.Elapsed.ToString())"
# bulk insert strings into temp table
$timer = [System.Diagnostics.Stopwatch]::StartNew()
$connection.Open();
$command = New-Object System.Data.SqlClient.SqlCommand("CREATE TABLE #temp_strings(StringValue varchar(20));", $connection)
[void]$command.ExecuteNonQuery()
$bcp = New-Object System.Data.SqlClient.SqlBulkCopy($connection)
$bcp.DestinationTableName = "#temp_strings"
$bcp.WriteToServer($dataTable)
$bcp.Close()
Write-Host "BCP completed. Duration $($timer.Elapsed.ToString())"
# execute set-based lookup query and return found/notfound for each string
$timer = [System.Diagnostics.Stopwatch]::StartNew()
$command.CommandText = #"
SELECT
strings.StringValue
, CASE
WHEN YourTable.YourTableKey IS NOT NULL THEN CAST(1 AS bit)
ELSE CAST(0 AS bit)
END AS Found
FROM #temp_strings AS strings
LEFT JOIN dbo.YourTable ON strings.StringValue = YourTable.YourTableKey;
"#
$reader = $command.ExecuteReader()
while($reader.Read()) {
Write-Host "String $($reader["StringValue"]) found: $($reader["Found"])"
}
$connection.Close()
Write-Host "Lookups completed. Duration $($timer.Elapsed.ToString())"
As an alternative to bulk insert, you could alternatively pass the strings using a table-valued parameter (or XML, JSON, delimited values) for use in the query.

Iterate through array of variables passing each one to a function in PowerShell

I have a function that exports the results of a SQL query to a json file:
# Connect to SQL Server
$SqlCommand.CommandText = $Query;
$SqlCommand.Connection = $SqlConnection;
# Execute query and get the result back
$QueryResult = $SqlCommand.ExecuteReader()
# Hold query result in data table
$QueryTable = New-Object "System.Data.DataTable"
$QueryTable.Load($QueryResult)
# Export query results to json
$QueryTable | Select-Object $QueryTable.Columns.ColumnName | ConvertTo-Json | Out-File "$OutputDirectory\$SqlInstance-$QueryName.json"
And I have multiple queries that I want to execute and have created variables for each one:
$q1 = "SELECT blah"
$q2 = "SELECT more blah"
$q3 = "SELECT even more blah"
I call the function by:
ExportQueryResultsToJson -Query $q1 -QueryName "q1"
I have around 80 queries that I want to execute so instead of having 80 lines of ExportQueryResultsToJson ... I want to use ForEach. I've created an array of variables:
$SqlServer2012QueryArray = #(
$q1,
$q2,
$q3
)
I've tried many variations of the following:
foreach ($Query in $SqlServer2012QueryArray) {
$Expression = "ExportQueryResultsToJson -Query '$Query' -QueryName $Query"
Invoke-Expression $Expresion
}
And I've tried using a splat but I can't figure out how to pass all queries in correctly.
What am I doing wrong?
You can approach this in a number of ways. Three possible ways, which are all very algorithmically similar, are below:
Using Your Array:
The solution depends on your array $sqlserver2012QueryArray having a list of sequentially numbered variables in the format q<number>. The first variable name must be q1.
for ($i = 0; $i -lt $sqlserver2012QueryArray.Count; $i++) {
ExportQueryResultsToJson -Query $sqlserver2012QueryArray[$i] -QueryName $((Get-Variable "q$($i+1)").Name)
}
Querying Already Created Variables:
This solution relies on your variables being named in the format q<number>. They do not have to be sequentially named. It could capture unwanted variables if they are named like q<number>abc.
foreach ($var in (Get-Variable -Name q[0-9]*)) {
ExportQueryResultsToJson -Query $var.Value -QueryName $var.Name
}
Using a Hash Table:
You can create a hash table with each key name being your variable name and the associated value being the query string. You can bypass creating the query variables all together with this solution by just inputting the query strings as the values.
$queryhash = #{'q1' = $q1; 'q2' = $q2; 'q3' = $q3; 'q14' = $q14}
foreach ($var in $queryhash.GetEnumerator()) {
ExportQueryResultsToJson -Query $var.Value -QueryName $var.Key
}
Note: In all cases, you should try to avoid Invoke-Expression. It is not generally a safe command to use because it welcomes code injection. I also don't see why it is necessary at all in this case either.

PowerShell execute SQL Query, Formatting output to excel. Losing some data from the output

I have a sql query I am running within Powershell, that exports the results to an excel workbook with multiple spreadsheets. I have most of my issues figured out now, however I have one rather large problem.
The result should return values that fill 3 columns. The query works outside of PowerShell and returns the expected result, but I am losing the results of the third column. It may be something obvious to the more seasoned, but I'm having a hard time.
I've been looking for a way to echo or output the result of my query to the screen from PowerShell to help me in determining where I am losing the data, but no dice yet.
Here is the script:
$DSN='mydsn'
$DirectoryToSave = 'c:\report\'
$SQL1=#"
My Working Query
"#
if (!(Test-Path -path "$DirectoryToSave")) #create it if not existing
{
New-Item "$DirectoryToSave" -type directory | out-null
}
$excel = New-Object -ComObject excel.application
$excel.visible = $True
$excel.DisplayAlerts = $False
$xl=$excel.Workbooks.Add()
$xl.Worksheets.Add()
$xl.Worksheets.Add()
$s1=$xl.sheets.Item(1)
$ws1 = $xl.sheets | where {$_.name -eq 'Sheet1'}
$ws1.name = $Project
$qt = $ws1.QueryTables.Add("ODBC;DSN=$DSN", $ws1.Range("A1"), $SQL1)
if ($qt.Refresh()){
$ws1.Activate()
$ws1.Select()
$excel.Rows.Item(1).HorizontalAlignment = $xlCenter
$excel.Rows.Item(1).VerticalAlignment = $xlTop
$excel.Rows.Item("1:1").Font.Name = "Calibri"
$excel.Rows.Item("1:1").Font.Size = 11
$excel.Rows.Item("1:1").Font.Bold = $true
$excel.Columns.item("A:C").EntireColumn.AutoFit()
$excel.Columns.item("B").NumberFormat=("$#,##0.00")
}
The values in the third colum should be numeric, and are normally returned as a 4 digit values separated by commas. each single cell can have an wide range of values returned. This is why I autofit with "$excel.Columns.item("A:C").EntireColumn.AutoFit()".
Anyone have any idea where I am going wrong?
Thanks In advance!
For now, wrap a CAST to VARCHAR around the STUFF part of the query from your previous question, i.e.:
SELECT
t.CUSTOMER,
SUM(t.REVENUE) as REVENUE,
CAST(
STUFF(
(
SELECT ',' + Cast(s.work_order as varchar)
FROM tblname s
WHERE s.CUSTOMER = t.CUSTOMER
FOR XML PATH('')
),1,1,''
) AS VARCHAR
) AS [WORK Order]
FROM tblname t
Group BY CUSTOMER
I can't explain why this works at the moment, but I suspect it's down to some string/character type or encoding funkyness going on in the ODBC driver. I'll try and delve deeper when I have more time.

Resources