Export array with "sub-array" - arrays

I am currently trying to automate license counting in Office 365 across multiple partner tenants using PowerShell.
My current code (aquired from the internet) with some modifications gives me this output:
Column A Column B Column C
-------- -------- --------
CustA LicA,LicB 1,3
CustB LicA,LicB 7,3
CustC LicA 4
But the output I want from this code is:
Column A Column B Column C
-------- -------- --------
CustA LicA 1
LicB 3
CustB LicA 7
LicB 3
Here is my current code which is exported using Export-Csv -NoType:
$tenantID = (Get-MsolPartnerContract).tenantid
foreach($i in $tenantID){
$tenantName = Get-MsolPartnerInformation -TenantId $i
$tenantLicense = Get-MsolSubscription -TenantId $i
$properties = [ordered]#{
'Company' = ($tenantName.PartnerCompanyName -join ',')
'License' = ($tenantLicense.SkuPartNumber -join ',')
'LicenseCount' = ($tenantLicense.TotalLicenses -join ',')
}
$obj = New-Object -TypeName psobject -Property $properties
Write-Output $obj
}
I have tried this along with several other iterations of code which all fail catastophically:
$properties = [ordered]#{
'Company' = ($tenantName.PartnerCompanyName -join ','),
#{'License' = ($tenantLicense.SkuPartNumber -join ',')},
#{'LicenseCount' = ($tenantLicense.TotalLicenses -join',')}
}
I was thinking about making a "sub-array" $tenantLicense.SkuPartnumber and $tenantLicense.TotalLicenses, but I'm not quite sure how to approach this with appending it to the object or "main-array".

A second loop for each $tenantLIcense should do the trick for you. I don't have access to an environment like yours so I cannot test this.
$tenantID | ForEach-Object{
$tenantName = Get-MsolPartnerInformation -TenantId $_
$tenantLicense = Get-MsolSubscription -TenantId $_
# Make an object for each $tenantLicense
$tenantLicense | ForEach-Object{
$properties = [ordered]#{
'Company' = $tenantName.PartnerCompanyName
'License' = $_.SkuPartNumber
'LicenseCount' = $_.TotalLicenses
}
# Send the new object down the pipe.
New-Object -TypeName psobject -Property $properties
}
}
Since you have multiple $tenantLicenses that have the same company name lets just loop over those and use the same company name in the output. Assuming this worked it would not have the same output as you desired since there no logic to omit company in subsequent rows. I would argue that is it better this way since you can sort the data now with out loss of data / understanding.
Notice I change foreach() to ForEach-Object. This makes it simpler to send object down the pipe.

Without providing the code solution I can say that you need to build up the array.
In terms of programming, you will need to iterate the array ARRAY1 and populate another one ARRAY2with the extra rows. For example if columns A,B are simple value and C is an array of 3 items, then you would add 3 rows in the new table with A,B,C1, A,B,C2 and A,B,C3. On each iteration of the loop you need to calculate all the permutations, for example in your case the ones generated by columnB and columnC.
This should be also possible with pipelining using the ForEach-Object cmdlet but that is more difficult and as you mentioned your relatively new relationship with powershell I would not pursuit this path, unless of coarse you want to learn.

Related

Query PSCustomObject Array for row with largest value

I'm trying to find the row with an attribute that is larger than the other row's attributes. Example:
$Array
Name Value
---- ----
test1 105
test2 101
test3 512 <--- Selects this row as it is the largest value
Here is my attempt to '1 line' this but It doesn't work.
$Array | % { If($_.value -gt $Array[0..($Array.Count)].value){write-host "$_.name is the largest row"}}
Currently it outputs nothing.
Desired Output:
"test1 is the largest row"
I'm having trouble visualizing how to do this efficiently with out some serious spaghetti code.
You could take advantage of Sort-Object to rank them by the property "Value" like this
$array = #(
[PSCustomObject]#{Name='test1';Value=105}
[PSCustomObject]#{Name='test2';Value=101}
[PSCustomObject]#{Name='test3';Value=512}
)
$array | Sort-Object -Property value -Descending | Select-Object -First 1
Output
Name Value
---- -----
test3 512
To incorporate your write host you can just run the one you select through a foreach.
$array | Sort-Object -Property value -Descending |
Select-Object -First 1 | Foreach-Object {Write-host $_.name,"has the highest value"}
test3 has the highest value
Or capture to a variable
$Largest = $array | Sort-Object -Property value -Descending | Select-Object -First 1
Write-host $Largest.name,"has the highest value"
test3 has the highest value
PowerShell has many built in features to make tasks like this easier.
If this is really an array of PSCustomObjects you can do something like:
$Array =
#(
[PSCustomObject]#{ Name = 'test1'; Value = 105 }
[PSCustomObject]#{ Name = 'test2'; Value = 101 }
[PSCustomObject]#{ Name = 'test3'; Value = 512 }
)
$Largest = ($Array | Sort-Object Value)[-1].Name
Write-host $Largest,"has the highest value"
This will sort your array according to the Value property. Then reference the last element using the [-1] syntax, then return the name property of that object.
Or if you're a purist you can assign the variable like:
$Largest = $Array | Sort-Object Value | Select-Object -Last 1 -ExpandProperty Name
If you want the whole object just remove .Name & -ExpandProperty Name respectively.
Update:
As noted PowerShell has some great tools to help with common tasks like sorting & selecting data. However, that doesn't mean there's never a need for looping constructs. So, I wanted to make a couple of points about the OP's own answer.
First, if you do need to reference array elements by index use a traditional For loop, which might look something like:
For( $i = 0; $i -lt $Array.Count; ++$i )
{
If( $array[$i].Value -gt $LargestValue )
{
$LargestName = $array[$i].Name
$LargestValue = $array[$i].Value
}
}
$i is commonly used as an iteration variable, and within the script block is used as the array index.
Second, even the traditional loop is unnecessary in this case. You can stick with the ForEach loop and track the largest value as and when it's encountered. That might look something like:
ForEach( $Row in $array )
{
If( $Row.Value -gt $LargestValue )
{
$LargestName = $Row.Name
$LargestValue = $Row.Value
}
}
Strictly speaking you don't need to assign the variables beforehand, though it may be a good practice to precede either of these with:
$LargestName = ""
$LargestValue = 0
In these examples you'd have to follow with a slightly modified Write-Host command
Write-host $LargestName,"has the highest value"
Note: Borrowed some of the test code from Doug Maurer's Fine Answer. Considering our answers were similar, this was just to make my examples more clear to the question and easier to test.
Figured it out, hopefully this isn't awful:
$Count = 1
$CurrentLargest = 0
Foreach($Row in $Array) {
# Compare This iteration vs the next to find the largest
If($Row.value -gt $Array.Value[$Count]){$CurrentLargest = $Row}
Else {$CurrentLargest = $Array[$Count]}
# Replace the existing largest value with the new one if it is larger than it.
If($CurrentLargest.Value -gt $Largest.Value){ $Largest = $CurrentLargest }
$Count += 1
}
Write-host $Largest.name,"has the highest value"
Edit: its awful, look at the other answers for a better way.

Efficient way to remove duplicates from large 2D arrays in PowerShell

I have a large set of data roughly 10 million items that I need to process efficiently and quickly removing duplicate items based on two of the six column headers.
I have tried grouping and sorting items but it's horrendously slow.
$p1 = $test | Group-Object -Property ComputerSeriaID,ComputerID
$p2 = foreach ($object in $p1.group) {
$object | Sort-Object -Property FirstObserved | Select-Object -First 1
}
The goal would be to remove duplicates by assessing two columns while maintaining the oldest record based on first observed.
The data looks something like this:
LastObserved : 2019-06-05T15:40:37
FirstObserved : 2019-06-03T20:29:01
ComputerName : 1
ComputerID : 2
Virtual : 3
ComputerSerialID : 4
LastObserved : 2019-06-05T15:40:37
FirstObserved : 2019-06-03T20:29:01
ComputerName : 5
ComputerID : 6
Virtual : 7
ComputerSerialID : 8
LastObserved : 2019-06-05T15:40:37
FirstObserved : 2019-06-03T20:29:01
ComputerName : 9
ComputerID : 10
Virtual : 11
ComputerSerialID : 12
You might want to clean up your question a little bit, because it's a little bit hard to read, but I'll try to answer the best I can with what I can understand about what you're trying to do.
Unfortunately, with so much data there's no way to do this quickly. String Comparison and sorting are done by brute force; there is no way to reduce the complexity of comparing each character in one string against another any further than measuring them one at a time to see if they're the same.
(Honestly, if this were me, I'd just use export-csv $object and perform this operation in excel. The time tradeoff to scripting something like this only once just wouldn't be worth it.)
By "Items" I'm going to assume that you mean rows in your table, and that you're not trying to retrieve only the strings in the rows you're looking for. You've already got the basic idea of select-object down, you can do that for the whole table:
$outputFirstObserved = $inputData | Sort-Object -Property FirstObserved -Unique
$outputLastObserved = $inputData | Sort-Object -Property LastObserved -Unique
Now you have ~20 million rows in memory, but I guess that beats doing it by hand. All that's left is to join the two tables. You can download that Join-Object command from the powershell gallery with Install-Script -Name Join and use it in the way described. If you want to do this step yourself, the easiest way would be to squish the two tables together and sort them again:
$output = $outputFirstObserved + $outputLastObserved
$return = $output | Sort-Object | Get-Unique
Does this do it? It keeps the one it finds first.
$test | sort -u ComputerSeriaID, ComputerID
I created this function to de-duplicate my multi-dimensional arrays.
Basically, I concatenate the contents of the record, add this to a hash.
If the concatenate text already exists in the hash, don't add it to the array to be returned.
Function DeDupe_Array
{
param
(
$Data
)
$Return_Array = #()
$Check_Hash = #{}
Foreach($Line in $Data)
{
$Concatenated = ''
$Elements = ($Line | Get-Member -MemberType NoteProperty | % {"$($_.Name)"})
foreach($Element in $Elements)
{
$Concatenated += $line.$Element
}
If($Check_Hash.$Concatenated -ne 1)
{
$Check_Hash.add($Concatenated,1)
$Return_Array += $Line
}
}
return $Return_Array
}
Try the following script.
Should be as fast as possible due to avoiding any pipe'ing in PS.
$hashT = #{}
foreach ($item in $csvData) {
# Building hash table key
$key = '{0}###{1}' -f $item.ComputerSeriaID, $item.ComputerID
# if $key doesn't exist yet OR when $key exists and "FirstObserverd" is less than existing one in $hashT (only valid when date provided in sortable format / international format)
if ((! $hashT.ContainsKey($key)) -or ( $item.FirstObserved -lt $hashT[$key].FirstObserved )) {
$hashT[$key] = $item
}
}
$result = $hashT.Values

PowerShell - need ini file for consecutive numbering

I'm currently working on a script for automation. This script should have a global count variable that does not reset itself when the script is executed again. Therefore, I need a configuration file that stores this count variable and uses it when it is called up again. This counting variable is also dependent on an ID. There is therefore a count variable for each ID. The configuration file can be in XML or INI format. Can someone tell me how to create such a file the easiest way and how to add IDs or get the count variable? I dont think "csv-import/export" is the right way.
I've already tried this...
$results = #()
$details = #{
Key1 = $ID
Key2 = $count
Key3 = "sth"
Key4 = "sth"
Key5 = "sth"
}
$results += New-Object PSObject -Property $details
$results | export-csv -Path C:\Users\...\configure.txt -NoTypeInformation
Unfortunately, I can't get any further here, because it overwrites the previous entry every time the ID changes and I don't know how to add additional entries (if the ID already exists), update entries (count variable) and call this count variable to use it in Powershell.
Anybody got a suggestion?
Best Regards
You can use a hash table, Export-CliXml and Import-CliXml to save and load you ID counts to a XML file:
$xmlFilePath = 'idCounts.xml'
# If the XML file exists, it is loaded
if( Test-Path -Path $xmlFilePath -PathType Leaf )
{
$hashTable = Import-Clixml -Path $xmlFilePath
}
# Else a new hash table is initialized
else
{
$hashTable = #{}
}
# Set the count of ID '001' to 1
$hashTable['001'] = 1
# Increment the count of ID '002'
$hashTable['002'] += 1
# Save the hash table to the XML file
$hashTable | Export-Clixml -Path $xmlFilePath
Thank you for all the tips. In the end, I managed it myself in the following way:
if(!((import-csv "C:\Users\...\Desktop\ini.txt") | where-object {$_.Key1 -eq $ID}))
{
$results = #()
$details = #{
Key 1 = $ID
Key 2 = 1
Key 3 = "something"
Key 4 = "something"
Key 5 = "something"
Key 6 = "something"
}
$results += New-Object PSObject -Property $details
$results | export-csv -Path C:\Users\...\Desktop\ini.txt -append -NoTypeInformation
}
The system first checks whether there is an entry with the corresponding ID. If not, an object is created that has that ID. The count variable is set to 1 when it is newly created. The entry is attached to the file with "Export CSV".
$select = (import-csv "C:\Users\...\Desktop\ini.txt" | where{$_.Key1 -eq $ID})
[int]$global:number = [convert]::ToInt32($select.Key2)
To use the count variable, the configuration file is imported. I have set it to "global" because it has to operate over several functions.
($csv = Import-Csv "C:\Users\...\Desktop\ini.txt") | ForEach {
if ($_.Key1 -eq $ID) {
$_.Key2 = $global:number}
}
$csv | Export-Csv "C:\Users\...\Desktop\ini.txt" -NoTypeInformation
At the end, the count variable is updated and transferred back to the file with "Export CSV".
Nevertheless thank you for all the interesting suggestions!
Best Regards

Powershell match properties and then selectively combine objects to create a third

I have a solution for this but I believe it is not the best method as it takes forever so I am looking for a faster/better/smarter way.
I have multiple pscustomObject objects pulled from .csv files. Each object has at least one common property. One is relatively small (around 200-300 items/lines in the object) but the other is sizable (around 60,000-100,000 items). The contents of one may or may not match the contents of the other.
I need to find where the two objects match on a specific property and then combine the properties of each object into one object with all or most properties.
An example snippet of the code (not exact but for this it should work - see the image for the sample data):
DataTables
Write-Verbose "Pulling basic Fruit data together"
$Purchase = import-csv "C:\Purchase.csv"
$Selling = import-csv "C:\Selling.csv"
Write-Verbose "Combining Fruit names and removing duplicates"
$Fruits = $Purchase.Fruit
$Fruits += $Selling.Fruit
$Fruits = $Fruits | Sort-Object -Unique
$compareData = #()
Foreach ($Fruit in $Fruits) {
$IndResults = #()
$IndResults = [pscustomobject]#{
#Adding Purchase and Selling data
Farmer = $Purchase.Where({$PSItem.Fruit -eq $Fruit}).Farmer
Region = $Purchase.Where({$PSItem.Fruit -eq $Fruit}).Region
Water = $Purchase.Where({$PSItem.Fruit -eq $Fruit}).Water
Market = $Selling.Where({$PSItem.Fruit -eq $Fruit}).Market
Cost = $Selling.Where({$PSItem.Fruit -eq $Fruit}).Cost
Tax = $Selling.Where({$PSItem.Fruit -eq $Fruit}).Tax
}
Write-Verbose "Loading Individual results into response"
$CompareData += $IndResults
}
Write-Output $CompareData
I believe the issue is in lines like these:
Farmer = $Purchase.Where({$PSItem.Fruit -eq $Fruit}).Farmer
If I understand this it is looking through the $Purchase object each time it goes through this line. I am looking for a way to speed that whole process up instead of having it look through the entire object for each match attempt.
Using this Join-Object:
$Purchase | Join $Selling -On Fruit | Format-Table
Result (using Simon Catlin's data):
Fruit Farmer Region Water Market Cost Tax
----- ------ ------ ----- ------ ---- ---
Apple Adam Alabama 1 MarketA 10 0.1
Cherry Charlie Cincinnati 2 MarketC 20 0.2
Damson Daniel Derby 3 MarketD 30 0.3
Elderberry Emma Eastbourne 4 MarketE 40 0.4
Fig Freda Florida 5 MarketF 50 0.5
using Join-Object
http://ramblingcookiemonster.github.io/Join-Object/
Join-Object -Left $purchase -Right $selling -LeftJoinProperty fruit -RightJoinProperty fruit -Type OnlyIfInBoth | ft
I had this very problem when trying to consolidate employee data from our HR system against employee data in our AD forest. With many thousands of rows, the process was taking an age.
I eventually walked away from custom objects and reverted to old school hash tables.
The hash tables entries themselves then held a sub-hash table with the data. In your instance, the outer hash would be keyed on $fruit, with the sub-hash containing the various attributes, e.g.: farmer, region, Etc.
Hash tables are lightning quick in comparison. It's a shame that PowerShell is slow in this regard.
Shout if you need more info.
26/01 Example code... assuming I'm correctly understanding the requirement:
PURCHASE.CSV:
Fruit,Farmer,Region,Water
Apple,Adam,Alabama,1
Cherry,Charlie,Cincinnati,2
Damson,Daniel,Derby,3
Elderberry,Emma,Eastbourne,4
Fig,Freda,Florida,5
SELLING.CSV
Fruit,Market,Cost,Tax
Apple,MarketA,10,0.1
Cherry,MarketC,20,0.2
Damson,MarketD,30,0.3
Elderberry,MarketE,40,0.4
Fig,MarketF,50,0.5
CODE
[String] $Local:strPurchaseFile = 'c:\temp\purchase.csv';
[String] $Local:strSellingFile = 'c:\temp\selling.csv';
[HashTable] $Local:objFruitHash = #{};
[System.Array] $Local:objSelectStringHit = $null;
[String] $Local:strFruit = '';
if ( (Test-Path -LiteralPath $strPurchaseFile -PathType Leaf) -and (Test-Path -LiteralPath $strSellingFile -PathType Leaf) ) {
#
# Populate data from purchase file.
#
foreach ( $objSelectStringHit in (Select-String -LiteralPath $strPurchaseFile -Pattern '^([^,]+),([^,]+),([^,]+),([^,]+)$' | Select-Object -Skip 1) ) {
$objFruitHash[ $objSelectStringHit.Matches[0].Groups[1].Value ] = #{ 'Farmer' = $objSelectStringHit.Matches[0].Groups[2].Value;
'Region' = $objSelectStringHit.Matches[0].Groups[3].Value;
'Water' = $objSelectStringHit.Matches[0].Groups[4].Value;
};
} #foreach-purchase-row
#
# Populate data from selling file.
#
foreach ( $objSelectStringHit in (Select-String -LiteralPath $strSellingFile -Pattern '^([^,]+),([^,]+),([^,]+),([^,]+)$' | Select-Object -Skip 1) ) {
$objFruitHash[ $objSelectStringHit.Matches[0].Groups[1].Value ] += #{ 'Market' = $objSelectStringHit.Matches[0].Groups[2].Value;
'Cost' = [Convert]::ToDecimal( $objSelectStringHit.Matches[0].Groups[3].Value );
'Tax' = [Convert]::ToDecimal( $objSelectStringHit.Matches[0].Groups[4].Value );
};
} #foreach-selling-row
#
# Output data. At this point, you could now build a PSCustomObject.
#
foreach ( $strFruit in ($objFruitHash.Keys | Sort-Object) ) {
Write-Host -Object ( '{0,-15}{1,-15}{2,-15}{3,-10}{4,-10}{5,10:C}{6,10:P}' -f
$strFruit,
$objFruitHash[$strFruit]['Farmer'],
$objFruitHash[$strFruit]['Region'],
$objFruitHash[$strFruit]['Water'],
$objFruitHash[$strFruit]['Market'],
$objFruitHash[$strFruit]['Cost'],
$objFruitHash[$strFruit]['Tax']
);
} #foreach
} else {
Write-Error -Message 'File error.';
} #else-if
I needed to do this myself for something similar. I wanted to take two system array objects and compare them pulling out the matches without having to manipulate the input data each time. Here's the method I used, which although I appreciate this is inefficient, it was instantaneous for the 200 or so records I had to work with.
I tried to translate what I was doing (users and their old and new home directories) into farmers, fruit and markets etc so I hope it makes sense!
$Purchase = import-csv "C:\Purchase.csv"
$Selling = import-csv "C:\Selling.csv"
$compareData = #()
foreach ($iPurch in $Purchase) {
foreach ($iSell in $Selling) {
if ($iPurch.fruit -match $iSell.fruit) {
write-host "Match found between $($iPurch.Fruit) and $($iSell.Fruit)"
$hash = #{
Fruit = $iPurch.Fruit
Farmer = $iPurch.Farmer
Region = $iPurch.Region
Water = $iPurch.Water
Market = $iSell.Market
Cost = $iSell.Cost
Tax = $iSell.Tax
}
$Build = New-Object PSObject -Property $hash
$Total = $Total + 1
$compareData += $Build
}
}
}
Write-Host "Processed $Total records"

Generically Transpose PowerShell Array Rows into Columns

There are quite a few posts on SO that address PowerShell transposition. However, most of the code is specific to the use case or addresses data being gathered from a text/CSV file and does me no good. I'd like to see a solution that can do this work without such specifics and works with arrays directly in PS.
Example data:
Customer Name: SomeCompany
Abbreviation: SC
Company Contact: Some Person
Address: 123 Anywhere St.
ClientID: XXXX
This data is much more complicated, but I can work with it using other methods if I can just get the rows and columns to cooperate. The array things that "Name:" and "SomeCompany" are column headers. This is a byproduct of how the data is gathered and cannot be changed. I'm importing the data from an excel spreadsheet with PSExcel and the spreadsheet format is not changeable.
Desired output:
Customer Name:, Abbreviation:, Company Contact:, Address:, ClientID:
SomeCompany, SC, Some Person, 123 Anywhere St., XXXX
Example of things I've tried:
$CustInfo = Import-XLSX -Path "SomePath" -Sheet "SomeSheet" -RowStart 3 -ColumnStart 2
$b = #()
foreach ($Property in $CustInfo.Property | Select -Unique) {
$Props = [ordered]#{ Property = $Property }
foreach ($item in $CustInfo."Customer Name:" | Select -Unique){
$Value = ($CustInfo.where({ $_."Customer Name:" -eq $item -and
$_.Property -eq $Property })).Value
$Props += #{ $item = $Value }
}
$b += New-Object -TypeName PSObject -Property $Props
}
This does not work because of the "other" data I mentioned. There are many other sections in this particular workbook so the "Select -Unique" fails without error and the output is blank. If I could limit the input to only select the rows/columns I needed, this might have a shot. It appears that while there is a "RowStart" and "ColumnStart" to Import-XLSX, there are no properties for stopping either one.
I've tried methods from the above linked SO questions, but as I said, they are either too specific to the question's data or apply to importing CSV files and not working with arrays.
I was able to resolve this by doing two things:
Removed the extra columns by using the "-Header" switch on the Import-XLSX function to add fake header names and then only select those headers.
$CustInfo = Import-XLSX -Path "SomePath" -Sheet "SomeSheet" -RowStart 2 -ColumnStart 2 -Header 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18 | Select "1","2"
The downside to this is that I had to know how many columns the input data had -- Not dynamic. If anyone can provide a solution to this issue, I'd be grateful.
Flipped the columns and headers with a simple foreach loop:
$obj = [PSCustomObject]#{}
ForEach ($item in $CustInfo) {
$value = $null
$name = $null
if ($item."2") { [string]$value = $item."2" }
if ($item."1") { [string]$name = $item."1" }
if ($value -and $name) {
$obj | Add-Member -NotePropertyName $name -NotePropertyValue $value
}
}
I had to force string type on the property names and values because the zip codes and CustID was formatting as an Int32. Otherwise, this does what I need.

Resources