Can a PS Custom Object be created from a variable? - arrays

Question

I have a 100 column table in sql server and I want to make it so not all of the columns need to be passed in the file to load. I have assigned column names in a table that then compares the columns in a hash table to find matching columns. I then create the code based on the match for the array I want to use to insert the data from the file. The problem is, it doesn't like calling the one variable to create the custom object.
I store the following below in a array. (up to a 100 of these, few below for sample (notice sqlcolumn2 is skipped for example)).
sqlcolumn1 = if ([string]::IsNullOrEmpty($obj.P1) -eq $true) {$null} else {"$obj.P1"}
sqlcolumn3 = if ([string]::IsNullOrEmpty($obj.P2) -eq $true) {$null} else {"$obj.P2"}
sqlcolumn4 = if ([string]::IsNullOrEmpty($obj.P3) -eq $true) {$null} else {"$obj.P3"}
sqlcolumn5 = if ([string]::IsNullOrEmpty($obj.P4) -eq $true) {$null} else {"$obj.P4"}
Here is the array:
foreach($line in $Final)
{
$DataRow = "$($line."TableColumnName") = if ([string]::IsNullOrEmpty(`$obj.$($line."PName")) -eq `$true) {`$null} else {`"`$obj.$($line."PName")`"}"
$DataArray += $DataRow
}
I then try to add it to a final array where I would want this to be looped through for each row of data after which I would perform the insert from the array. Even though the "string" value in the array above is correct if it were hand coded, I can't get it to recognize the rows and run.
foreach ($obj in $data2)
{
$test = [PSCustomObject] #{
$DataArray = Invoke-Expression $DataArray
}
If I just type $DataArray, it doesn't like this because it wants the = sign which I already have built into the string.
Is what I am trying to do even possible.
I was attempting to template out various different ways we receive this data, where some people send us 30 of the 100 columns, other more or less, and no one person using the exact columns to cut down on individual scripts for everything.
Adding more code:
Function ArrayCompare() {
[CmdletBinding()]
PARAM(
[Parameter(Mandatory=$True)]$Array1,
[Parameter(Mandatory=$True)]$A1Match,
[Parameter(Mandatory=$True)]$Array2,
[Parameter(Mandatory=$True)]$A2Match)
$Hash = #{}
foreach ($Data In $Array1) {
$Hash[$Data.$A1Match] += ,$Data
}
foreach ($Data In $Array2) {
$Hash[$Data.$A2Match] += ,$Data
}
foreach ($KeyValue In $Hash.GetEnumerator()){
$Match1, $Match2 = $KeyValue.Value.Where( {$_.$A1Match}, 'Split')
[PSCustomObject]#{
MatchValue = $KeyValue.Key
A1Matches = $Match1.Count
A2Matches = $Match2.Count
TablePosition = [int]$Match2.TablePosition
TableColumnName = $Match2.TableColumnName
# PName is the P(##) that is a generic ascending column value back to import-excel module. ColumnA = P1, ColumnB = P2 etc..until no data is detected. Allows flexibility and not having to know how many columns there are
PName = $Match1.Name}
}
}
$Server = 'ServerName'
$Catalog = 'DBName'
$DestinationTable = 'ImportIntoTableName'
$FileIdentifierID = 10
$FileName = 'Test.xlsx'
$FilePath = 'C:\'
$FullFilePath = $FilePath + $FileName
$data = Import-Excel -Path $FullFilePath -NoHeader -StartRow 1 # Import-
Excel Module for working with xlsx excel files
$data2 = Import-Excel -Path $ullFilePath -NoHeader -StartRow 2 # Import-
Excel Module for working with xlsx excel files
$ExpectedHeaderArray = #()
$HeaderArray = #()
$DataArray = #()
$HeaderDetect = #()
$HeaderDetect = $data | Select-Object -First 1 # Header Row In File
$HeaderDetect |
ForEach-Object {
$ColumnValue = $_
$ColumnValue |
Get-Member -MemberType *Property |
Select-Object -ExpandProperty Name |
ForEach-Object {
$HeaderValues = [PSCustomObject]#{
Name = $_
Value = $ColumnValue.$_}
$HeaderArray += $HeaderValues
}
}
# Query below provides a list of all expected file headers and the table
column name they map to
$Query = "SELECT TableColumnName, FileHeaderName, TablePosition FROM
dbo.FileHeaders WHERE FileIdentifierID = $($FileIdentifierID)"
$ds = Invoke-Sqlcmd -ServerInstance $Server -Database $Catalog -Query $Query
-OutputAs DataSet
$ExpectedHeaderArray = foreach($Row in $ds.Tables[0].Rows)
{
new-object psObject -Property #{
TableColumnName = "$($row.TableColumnName)"
FileHeaderName = "$($row.FileHeaderName)"
TablePosition = "$($row.TablePosition)"
}
}
#Use Function Above
#Bring it together so we know what P(##) goes with which header in file/mapped to table column name
$Result = ArrayCompare -Array1 $HeaderArray -A1Match Value -Array2 $ExpectedHeaderArray -A2Match FileHeaderName
$Final = $Result | sort TablePosition
foreach($Line in $Final)
{
$DataRow = "$($Line."TableColumnName") = if ([string]::IsNullOrEmpty(`$obj.$($Line."PName")) -eq `$true) {`$null} else {`"`$obj.$($Line."PName"))`"}"
$DataArray += $DataRow
}
# The output below is what the code inside the last array would be that I would use to import into excel.
# The goal is to be dynamic and match headers in the file to the stored header value and import into a table (mapped from header column to table column name)
# The reason for this is before I was here, there were many different "versions" of a layout that was given out. In the end, it is all one in the same
# but some send all 100 columns, some only send a handful, some send 80 etc. I am trying to have everything flow through here vs. 60+ pieces of code/stored procedures/ssis packs
Write-Output $DataArray
# Output Sample -- Note how in the sample, P2 and subsequent skip SQLColumn2 because P2 maps to the header value of position 3 in the sql table and each after is one off.
# In this example, SqlColumn2 would not be populated
# SqlColumn1 = if ([string]::IsNullOrEmpty($obj.P1) -eq $true) {$null} else {"$obj.P1"}
# SqlColumn3 = if ([string]::IsNullOrEmpty($obj.P2) -eq $true) {$null} else {"$obj.P2"}
# SqlColumn4 = if ([string]::IsNullOrEmpty($obj.P3) -eq $true) {$null} else {"$obj.P3"}
# SqlColumn5 = if ([string]::IsNullOrEmpty($obj.P4) -eq $true) {$null} else {"$obj.P4"}
# I know this doesn't work. This is where I'm stuck, how to build an array now off of this output from above
foreach ($obj in $data2)
{
$test = [PSCustomObject] #{
$DataArray = Invoke-Expression $DataArray}
}

Answer 1

I'm gong to re-state your question first, just to make sure I understand it properly (it's possible I don't!)...
You've got an excel file that looks something like this:
+---+---------+---------+---------+
| | A | B | C |
+---+---------+---------+---------+
| 1 | HeaderA | HeaderB | HeaderC |
+---+---------+---------+---------+
| 2 | Value P | Value Q | Value R |
+---+---------+---------+---------+
| 3 | Value S | Value T | Value U |
+---+---------+---------+---------+
You've also got a database table which looks like this:
+---------+---------+---------+---------+
+ ColumnW | ColumnX | ColumnY | ColumnZ |
+---------+---------+---------+---------+
+ ....... | ....... | ....... | ....... |
+---------+---------+---------+---------+
and a column mapping table like this (note, ColumnX isn't mapped in this example):
+-----------------+----------------+---------------+
| TableColumnName | FileHeaderName | TablePosition |
+-----------------+----------------+---------------+
| ColumnW | HeaderA | 1 |
+-----------------+----------------+---------------+
| ColumnY | HeaderB | 2 |
+-----------------+----------------+---------------+
| ColumnZ | HeaderC | 3 |
+-----------------+----------------+---------------+
You want to insert the values from the spreadsheet into the database table, using the data in your mapping table so you get this:
+---------+---------+---------+---------+
+ ColumnW | ColumnX | ColumnY | ColumnZ |
+---------+---------+---------+---------+
+ Value P | null | Value Q | Value R |
+---------+---------+---------+---------+
+ Value S | null | Value T | Value U |
+---------+---------+---------+---------+
So let's load the spreadsheet (letting the header row generate meaningful property names this time):
$data = Import-Excel -Path ".\MySpreadsheet.xlsx";
write-host ($data | ft | out-string);
# HeaderA HeaderB HeaderC
# ------- ------- -------
# Value P Value Q Value R
# Value S Value T Value U
and get your column mapping data (I'm programmatically creating an in-memory dataset, but you obviously read yours from your database instead):
$mappings = new-object System.Data.DataTable;
$null = $mappings.Columns.Add("TableColumnName", [string]);
$null = $mappings.Columns.Add("FileHeaderName", [string]);
$null = $mappings.Columns.Add("TablePosition", [int]);
#(
#{ "TableColumnName"="ColumnW"; "FileHeaderName"="HeaderA"; "TablePosition"=1 },
#{ "TableColumnName"="ColumnY"; "FileHeaderName"="HeaderB"; "TablePosition"=2 },
#{ "TableColumnName"="ColumnZ"; "FileHeaderName"="HeaderC"; "TablePosition"=3 }
) | % {
$row = $mappings.NewRow();
$row.TableColumnName = $_.TableColumnName;
$row.FileHeaderName = $_.FileHeaderName;
$row.TablePosition = $_.TablePosition;
$mappings.Rows.Add($row);
}
$ds = new-object System.Data.DataSet;
$ds.Tables.Add($mappings);
write-host ($ds.Tables[0] | ft | out-string)
# TableColumnName FileHeaderName TablePosition
# --------------- -------------- -------------
# ColumnW HeaderA 1
# ColumnY HeaderB 2
# ColumnZ HeaderC 3
Now we can build the "mapped" objects:
$values = #();
foreach( $row in $data )
{
$properties = [ordered] #{};
foreach( $mapping in $mappings )
{
$properties.Add($mapping.TableColumnName, $row."$($mapping.FileHeaderName)");
}
$values += new-object PSCustomObject -Property $properties;
}
write-host ($values | ft | out-string)
# ColumnW ColumnY ColumnZ
# ------- ------- -------
# Value P Value Q Value R
# Value S Value T Value U
The tricksy bit is $properties.Add($mapping.TableColumnName, $row."$($mapping.FileHeaderName)"); - basically, you can access object properties in PowerShell using a dotted string literal or variable (I'm not sure of the exact feature name) - e.g.
PS> $myValue = new-object PSCustomObject -Property #{ "aaa"="bbb"; "ccc"="ddd" }
PS> $myValue."aaa"
bbb
PS> $myProperty = "aaa"
PS> $myValue.$myProperty
"bbb"
so $row."$($mapping.FileHeaderName)" is an expression that evaluates to the value of the property of $row named in $mapping.FileHeaderName.
And then finally you can insert the objects into your database using your existing process...
Note that I couldn't quite work out what your ArrayCompare is actually doing so it's possible the above doesn't solve your problem 100%, but it's hopefully close enough that you can either work the difference out yourself, or leave a comment with where it differs from your desired solution.
Hope this helps.