I've been trying to write a utility function that will return initialized [System.Data.DataTable] object data type.
I am aware of the annoying PowerShell functions return behavior where it tries to "unroll" everything into a [System.Array] return type.
Previously I have always managed to get around that problem.
Usually using the "comma trick" to return your own array #(,$result) aways worked - however this time this does not seem to make a difference and I've run out of options...
Another trick I normally use is the $null assignments within the Process block (see my code bellow) - this way I fool the pipeline there is nothing to "unroll" on the output...
I am not taking impossible for an answer, so far based on my experience nothing is impossible in PowerShell :)
Here is my code:
function Get-SourceDataTable
{
[OutputType([System.Data.DataTable])]
[cmdletbinding()]
param(
[parameter(Mandatory=$true, Position=0)]
[System.Data.SqlClient.SqlBulkCopy] $Destination,
[parameter(Mandatory=$true, Position=1)]
[System.Collections.Specialized.OrderedDictionary] $ColumnDefinition,
[parameter(Mandatory=$false, Position=2)]
[int]$ColumnStartIndex = 0
)
BEGIN{
$datatable = New-Object System.Data.DataTable
$colIndex = $ColumnStartIndex
}
PROCESS{
$ColumnDefinition.Keys |
foreach {
$null = $datatable.Columns.Add($_, $ColumnDefinition[$_]) # define each column name and data type
$null = $Destination.ColumnMappings.Add($_, $colIndex) # map column to destination table
$colIndex++
}
}
END{
return ,$datatable
}
}
I hope someone can get this code working...
Rather than return use Write-Output -NoEnumerate. For example:
function New-DataTable {
$datatable = New-Object System.Data.DataTable
$null = $datatable.Columns.Add("x",[int])
$null = $datatable.Columns.Add("y",[int])
$null = $datatable.Rows.Add(#(1,2))
$null = $dataTable.Rows.Add(#(3,4))
Write-Output -NoEnumerate $datatable
}
New-DataTable | Get-Member
Note however that if you just type New-DataTable, it might look like enumberated rows, but Get-Member tells you the actual type returned.
I got the function from the question to return DataTable type when I used LoadWithPartialName to load the assembly containing the type and pipe it out with Out-Null.
Don't ask my why, but feel free to comment if you know the reason.
The working function code is below. Note the return statement is not necessary I only use it to improve code readability:
function Get-SourceDataTable
{
[OutputType([System.Data.DataTable])]
[cmdletbinding()]
param(
[parameter(Mandatory=$true, Position=0)]
[System.Data.SqlClient.SqlBulkCopy] $Destination,
[parameter(Mandatory=$true, Position=1)]
[System.Collections.Specialized.OrderedDictionary] $ColumnDefinition,
[parameter(Mandatory=$false, Position=2)]
[int]$ColumnStartIndex = 0
)
BEGIN{
[System.Reflection.Assembly]::LoadWithPartialName("System.Data") | Out-Null
$datatable = New-Object System.Data.DataTable
$colIndex = $ColumnStartIndex
}
PROCESS{
$ColumnDefinition.Keys |
foreach {
$null = $datatable.Columns.Add($_, $ColumnDefinition[$_]) # define each column name and data type
$null = $Destination.ColumnMappings.Add($_, $colIndex) # map column to destination table
$colIndex++
}
}
END{
return ,$datatable
}
}
To summarize all known possible solutions to the problem of forcing PowerShell function to return specific data type:
use $null assignments
use comma to return an array ,$variable
use LoadWithPartialName("Assembly.Name") | Out-Null
use Write-Output -NoEnumerate $variable to return the type - credit goes to Burt_Harris
Finally, after the input from Burt_Harris (THANKS Burt!) the final working version of the function from this question is this:
function Get-SourceDataTable
{
[OutputType([System.Data.DataTable])]
[cmdletbinding()]
param(
[parameter(Mandatory=$true, Position=0)]
[System.Data.SqlClient.SqlBulkCopy] $Destination,
[parameter(Mandatory=$true, Position=1)]
[System.Collections.Specialized.OrderedDictionary] $ColumnDefinition,
[parameter(Mandatory=$false, Position=2)]
[int]$ColumnStartIndex = 0
)
BEGIN{
#[System.Reflection.Assembly]::LoadWithPartialName("System.Data") | Out-Null
$datatable = New-Object System.Data.DataTable
$colIndex = $ColumnStartIndex
}
PROCESS{
$ColumnDefinition.Keys |
foreach {
$null = $datatable.Columns.Add($_, $ColumnDefinition[$_]) # define each column name and data type
$null = $Destination.ColumnMappings.Add($_, $colIndex) # map column to destination table
$colIndex++
}
}
END{
#return ,$datatable
Write-Output -NoEnumerate $datatable
}
}
Related
I have a function that replaces PackageID in a SCCM task sequence, I would like to capture all those package IDs into a variable, so I would be able to create a report based on that.
The problem is that I already have a foreach loop doing the work, and I can't figure out how to not overwrite the values.
$Driver.PackageID comes from a foreach loop based on $Drivers, which contains
If I run the code I get this as I have Write-Output defined:
Updated code:
function Set-Drivers{
foreach ($Driver in $Drivers) {
Write-Output "Driver Name: $($Driver.Name)"
Write-Output "DriverPackageID: $($Driver.PackageID)"
}
}
$array = #()
$array = Set-Drivers
$hash = [ordered]#{
'DriverName' = $Driver.Name
'DriverID' = $Driver.PackageID
}
$array += New-Object -Typename PSObject -Property $hash
Can someone explain, why I only get the first result in my $array? I can see the values are being overwritten if I run it in debug mode.
Your code is not iterating over the results, but instead only using one of them. This what you intended.
$array = $drivers | foreach {
[ordered]#{
DriverName = $_.Name
DriverID = $_.PackageID
}
}
Your function doesn't return anything. It only writes lines to the console. Then after the function is finished, you create a single object and add that to your array.
Try something like
function Set-Drivers{
$result = foreach ($Driver in $Drivers) {
[PsCustomObject]#{
'DriverName' = $Driver.Name
'DriverID' = $Driver.PackageID
}
}
# output the result
# the comma wraps the result in a single element array, even if it has only one element.
# PowerShell 'flattens' that upon return from the function, leaving the actual resulting array.
,$result
}
$array = Set-Drivers
# show what you've got
$array
I have a PowerShell script, where I want to make sure certain variables have value before proceeding.
So I have the following:
$dataRow = $sheet.Cells.Find($country).Row
$serverCol = $sheet.Cells.Find($serverString).Column
$databaseCol = $sheet.Cells.Find($databaseString).Column
$userCol = $sheet.Cells.Find($userString).Column
$passwordCol = $sheet.Cells.Find($passString).Column
$partnerCol = $sheet.Cells.Find($partnerString).Column
#All variables in this array are required. If one is empty - the script cannot continue
$requiredVars = #($dataRow, $serverCol, $databaseCol, $userCol, $passwordCol, $partnerCol)
But when I foreach over the array like so:
foreach ($var in $requiredVars)
{
Write-Host DataRow = ($dataRow -eq $var)
Write-Host ServerCol = ($serverCol -eq $var)
Write-Host DatabaseCol = ($databaseCol -eq $var)
Write-Host UserCol = ($userCol -eq $var)
Write-Host PasswordCol = ($passwordCol -eq $var)
Write-Host PartnerCol = ($partnerCol -eq $var)
if ($var -eq $null)
{
[System.Windows.Forms.MessageBox]::Show("No data found for given string!")
$excel.Quit()
return
}
}
I always get the MessageBox. I added the "Write-Host" part to see the value of each variable, then changed it to see which variable was null but all variables have values in them and all the checks you see here return "False".
I'd like to know what I'm doing wrong and if the $requiredVars array only copies values, not references or something.
Instead of using separate variables, you may consider using a Hashtable to store them all.
This makes checking the individual items a lot simpler:
# get the data from Excel and store everything in a Hashtable
# to use any of the items, use syntax like $excelData.passwordCol or $excelData['passwordCol']
$excelData = #{
'dataRow' = $sheet.Cells.Find($country).Row
'serverCol' = $sheet.Cells.Find($serverString).Column
'databaseCol' = $sheet.Cells.Find($databaseString).Column
'userCol' = $sheet.Cells.Find($userString).Column
'passwordCol' = $sheet.Cells.Find($passString).Column
'partnerCol' = $sheet.Cells.Find($partnerString).Column
}
# check all items in the hash. If any item is $null then exit
foreach ($item in $excelData.Keys) {
# or use: if ($null -eq $excelData[$item])
if (-not $excelData[$item]) {
[System.Windows.Forms.MessageBox]::Show("No data found for item $item!")
$excel.Quit()
# IMPORTANT: clean-up used COM objects from memory when done with them
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($sheet) | Out-Null
# Your code doesn't show this, but you'll have a $workbook object in there too
# [System.Runtime.Interopservices.Marshal]::ReleaseComObject($workbook) | Out-Null
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel) | Out-Null
[System.GC]::Collect()
[System.GC]::WaitForPendingFinalizers()
return
}
}
One way to directly solve your question is this:
$a = "foo"
$b = "bar"
$c = $null
$requiredVariables = $a, $b, $c
# How many total entries in array?
($requiredVariables).Count
# How many of them have a value?
($requiredVariables | Where-Object {$_}).Count
# So one option for a single check would be:
if (($requiredVariables.Count) -ne ($requiredVariables | Where-Object {$_}).Count) {
Write-Warning "Not all values provided"
}
However an alternative [and better] approach is to make your code in to a function that includes parameter validation
function YourCustomFunction {
Param (
[ValidateNotNullOrEmpty()]
$a
,
[ValidateNotNullOrEmpty()]
$b
,
[ValidateNotNullOrEmpty()]
$c
)
Process {
Write-Output "Your function code goes here..."
}
}
# Call your function with the params
YourCustomFunction -a $a -b $b -c $c
Example output:
Test-YourCustomFunction: Cannot validate argument on parameter 'c'. The argument is null or empty. Provide an argument that is not null or empty, and
then try the command again.
At line:39 char:48
This is driving me crazy. I have a library I source from multiple scripts, which contains the following function:
function lib_open_dataset([string] $sql) {
$ds = new-object "System.Data.DataSet"
$da = new-object "System.Data.SqlClient.SqlDataAdapter" ($sql, $_conn_string)
$record_count = $da.Fill($ds)
return $ds
}
This is called pretty much everywhere and it works just fine, except that I normally have to do this:
$ds = lib_open_dataset($some_sql)
$table = $ds.Tables[0]
foreach ($row in $table.Rows) {
# etc
}
So I created a new simple wrapper function to avoid the extra step of dereferencing the first table:
function lib_open_table([string] $sql) {
$ds = lib_open_dataset $sql
return $ds.Tables[0]
}
The problem is that what's being returned from here is the Rows collection of the table for some reason, not the table itself. This causes the foreach row loop written as above to fail with a "Cannot index into a null array." exception. After much trial and error I figured out this works:
foreach ($row in $table) {
# etc
}
Note the difference between $table.Rows and just $table in the foreach statement. This works. Because $table actually points to the Rows collection. If the statement
return $ds.Tables[0]
is supposedly correct, why is the function returning a child collection of the table object instead of the table itself?
I'm guessing there's something in the way Powershell functions work that's causing this obviously, but I can't figure out what.
You can use the comma operator to wrap the rows collection in an array so that when the array is unrolled you wind up with the original rows collection e.g.:
function lib_open_table([string] $sql) {
$ds = lib_open_dataset $sql
return ,$ds.Tables[0]
}
Essentially you can't prevent PowerShell from unrolling arrays/collections. The best you can do is workaround that behavior by wrapping the array/collection within another, single element array.
PowerShell special-cases the DataTable internally. It does not implement any of the usual suspect interfaces like ICollection, IList or IEnumerable which normally trigger the unrolling. You can dig into this a bit with:
PS> $dt = new-object data.datatable
PS> $dt -is [collections.ienumerable]
False
Yet:
PS> $e = [management.automation.languageprimitives]::GetEnumerator($dt)
PS> $e.gettype()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
False False RBTreeEnumerator System.ValueType
-Oisin
2 things you need to indeed focus on
a) prepend your returned object with the comma indeed
b) when you're filling your adaptor, make sure to either assign the outcome to a (disposalble) variable or do an Out-Null
I didn't do the Out-Null and even with a prepended comma, I kept getting a collection back (item 0= number of rows from the query, item1= the datatable)
Drove my a bit crazy until I picked the Out-null parameter out.
Very weird IMHO, as I'm asking specifically to return the datatable but kept getting the collection back, even with the "," in front
function Oracleconnection
{
process
{
trap
{
Write-Host "error occured on oracle connection"
Write-Host $_
continue
}
[System.Reflection.Assembly]::LoadWithPartialName(“System.Data.OracleClient”) | out-null
$connection = new-object system.data.oracleclient.oracleconnection( `
"Data Source=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myhost.host)(PORT=1800)) `
(CONNECT_DATA=(SERVICE_NAME=myservicename)));User Id=myid;Password=mypassword;");
$query = "SELECT country, asset FROM table "
$set = new-object system.data.dataset
$adapter = new-object system.data.oracleclient.oracledataadapter ($query, $connection)
$adapter.Fill($set) | Out-Null
$table = new-object system.data.datatable
$table = $set.Tables[0]
return ,$table
}
}
(Concepts from the Answer by Keith!)
I'm completely lost in a mind loop at the moment:
I wanted to solve the problem of passing an array to a function by value instead of reference and found this solution:
Is Arraylist passed to functions by reference in PowerShell.
The .Clone()-method works as expected:
function testlocal {
param ([collections.arraylist]$local)
$local = $local.Clone()
$local[0] = 10
return $local
}
$local = [collections.arraylist](1,2,3)
'Testing function arraylist'
$copyOfLocal = testlocal $local
$copyOfLocal
'Testing local arraylist'
$local
Output:
Testing function arraylist
10
2
3
Testing local arraylist
1
2
3
But now I need to process the array's elements in a foreach-loop. What happens then is that the array does not get modified by the foreach-loop (???). I am at a loss to understand this, despite a lot of research. Could you please explain to me what is happening behind the scenes and how I can avoid this?
I need to modify a copy of the original array within a function's foreach-loop. In my real script, the array consists of custom PSObjects, but the behavior is the same.
function testlocal {
param ([collections.arraylist]$local)
$local = $local.Clone()
$local[0] = 10
foreach ($item in $local) {
$item = 100
}
return $local
}
$local = [collections.arraylist](1,2,3)
'Testing function arraylist'
$copyOfLocal = testlocal $local
$copyOfLocal
'Testing local arraylist'
$local
Output is not changed by the foreach-loop:
Testing function arraylist
10
2
3
Testing local arraylist
1
2
3
UPDATE 2016-12-14
The tip with the for-loop works, but it turns out when using objects, the whole cloning-thing falls apart again:
function testlocal {
param ([collections.arraylist]$local)
$local = $local.Clone()
for($i = 0; $i -lt $local.Count; $i++){
$local[$i].Hostname = "newname"
}
return $local
}
$target1 = New-Object -TypeName PSObject
$target1 | Add-Member -MemberType NoteProperty -Name "Hostname" -Value "host1"
$target2 = New-Object -TypeName PSObject
$target2 | Add-Member -MemberType NoteProperty -Name "Hostname" -Value "host2"
$local = [collections.arraylist]($target1,$target2)
'Testing function arraylist'
$copyOfLocal = testlocal $local
$copyOfLocal | ft
'Testing local arraylist'
$local | ft
Output:
Testing function arraylist
Hostname
--------
newname
newname
Testing local arraylist
Hostname
--------
newname
newname
Suddenly I am back to passing by reference again. This is driving me mad!
Please help!
When you enumerate the array by calling foreach, you are getting copies of the content and any changes will get discarded. As Mathias Jessen mentioned you can use a for loop to make changes to the arraylist.
EDIT 2016-12-16
Okay, I looked into it and the arraylist clone method does not work like you and I thought it does. According to MSDN, it returns a shallow copy, i.e. it "copies only the elements of the collection, whether they are reference types or value types, but it does not copy the objects that the references refer to". You and I where both assuming that clone would produce what is called a "deep copy".
In your original example using the basic datatype Int32, deep and shallow copy are the same thing, while in your updated example using PSObjects, they are not and only the references are copied. A method to deep clone objects is found in this thread.
I did not find documentation on this, but it seems as if foreach would do shallow copies also:
$object1 = New-Object -TypeName PSObject -Property #{Hostname="host1"}
$object2 = New-Object -TypeName PSObject -Property #{Hostname="host2"}
$array_of_objects = [collections.arraylist]($object1,$object2)
# Looping through basic values, call by value
foreach ($string in $array_of_objects.Hostname)
{
$string = "newname"
}
$array_of_objects.Hostname
# Looping through objects, call by reference
foreach ($object in $array_of_objects)
{
$object.Hostname = "newname"
}
$array_of_objects.Hostname
Hope this helps.
I feel like perhaps I am overlooking something simple here, but I am having trouble with a ForEach loop in PowerShell actually returning all items that I expect. I have a script that will query an Oracle database and gather up the base data set. Once this is gathered, I will need to perform some adjustments to what is returned and build an additional bit of information (not in the script currently, working through the basics so far)
What I am doing is adding the data to an array, then trying to use a ForEach loop to examine each item in the array and pump that data out to another array that will have the new properties that I need to populate based on some calculations of the base data set. What I am getting returned to the variable $finaloutput is only one line of the data (for the example I am posting here I simply look for one reportnumber equal to CPOD-018, which there are 5 of in the data set with varying other properties populated including the sitename which I am populate as well, but still only get one result).
I've tried going about this using nested if statements within the ForEach loop instead of the piped Where-Object, but received the same results. Below is the current version of the script, any assistance would be greatly appreciated.
param(
[parameter(mandatory=$True)]$username,
[parameter(mandatory=$True)]$password
)
# setup the finaloutput variable
$finaloutput = New-Object psobject
$finaloutput | Add-Member -MemberType NoteProperty -name ReportNumber -value NotSet
$finaloutput | Add-Member -MemberType NoteProperty -name sitename -value NotSet
# the connection string to be used by the OlEDB connection
$connString = #"
Provider=OraOLEDB.Oracle;Data Source=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST="host.host.host")(PORT="1521"))
(CONNECT_DATA=(SERVICE_NAME="name.name.name")));User ID="$username";Password="$password"
"#
# the query that will be used to gather data from Oracle
$qry= #"
select VP_EXPECTED_DETAILS.REPORTNUMBER, VP_EXPECTED_DETAILS.SITE_NAME, VP_ACTUAL_FILENAME_DETAILS.FILE_NAME, VP_EXPECTED_DETAILS.MAX_EXPECTED_LOAD_TIME, VP_EXPECTED_DETAILS.EXPECTED_FREQUENCY,
VP_EXPECTED_DETAILS.DATE_TIMING, VP_EXPECTED_DETAILS.FREQUENCY_DAY, VP_EXPECTED_DETAILS.JOB_NO,
TO_CHAR(VP_ACTUAL_RPT_DETAILS.GEN_PARSE_IN,'YYYYMMDDHH24MISS') AS GEN_PARSE_IN,
TO_CHAR(VP_ACTUAL_RPT_DETAILS.GEN_PARSE_OUT,'YYYYMMDDHH24MISS') AS GEN_PARSE_OUT,
TO_CHAR(VP_ACTUAL_RPT_DETAILS.ETLLOADER_IN,'YYYYMMDDHH24MISS') AS ETLLOADER_IN,
TO_CHAR(VP_ACTUAL_RPT_DETAILS.ETLLOADER_OUT,'YYYYMMDDHH24MISS') AS ETLLOADER_OUT
from MONITOR.VP_EXPECTED_DETAILS
LEFT JOIN MONITOR.VP_ACTUAL_RPT_DETAILS on VP_EXPECTED_DETAILS.REPORTNUMBER = VP_ACTUAL_RPT_DETAILS.REPORTNUMBER and VP_EXPECTED_DETAILS.SITE_NAME = VP_ACTUAL_RPT_DETAILS.SITE_NAME
LEFT JOIN MONITOR.VP_ACTUAL_FILENAME_DETAILS on VP_ACTUAL_RPT_DETAILS.FNKEY = VP_ACTUAL_FILENAME_DETAILS.FNKEY where VP_EXPECTED_DETAILS.EXPECTED_FREQUENCY = 'DAILY' or
(VP_EXPECTED_DETAILS.EXPECTED_FREQUENCY = 'MONTHLY' AND VP_EXPECTED_DETAILS.FREQUENCY_DAY = EXTRACT(DAY from SYSDATE))
"#
# the function that will open the database connection and execute the query
function Get-OLEDBData ($connectstring, $sql) {
$OLEDBConn = New-Object System.Data.OleDb.OleDbConnection($connectstring)
$OLEDBConn.open()
$readcmd = New-Object system.Data.OleDb.OleDbCommand($sql,$OLEDBConn)
$readcmd.CommandTimeout = '300'
$da = New-Object system.Data.OleDb.OleDbDataAdapter($readcmd)
$dt = New-Object System.Data.DataTable
[void]$da.fill($dt)
$OLEDBConn.close()
return $dt
}
# populate $output with the data from the Get-OLEDBData function
$output = Get-OLEDBData $connString $qry
# build the final output that will generate alerts
ForEach ($lines in $output | Where-Object {$_.reportnumber -eq "CPOD-018"})
{
$finaloutput.reportnumber = $lines.reportnumber
$finaloutput.sitename = $lines.SITE_NAME
}
$finaloutput
It looks like what is happening is you are doing the object creation incorrectly. Inside the foreach loop you keep overwriting the same values rather than appending new objects
It should look more like this:
$FinalOutput = ForEach ($lines in $output | Where-Object {$_.reportnumber -eq "CPOD-018"})
{
$Prop = #{
'reportnumber' = $lines.reportnumber
'sitename' = $lines.SITE_NAME
}
New-Object -Type PSObject -Property $Prop
}
$FinalOutput
You would need to comment out the finaloutput lines earlier in your script.