Transpose to object but without values, yet - arrays

I'm importing esxtop data from an ESXi host. The import gives me two sets of data. The first is the list of all the fields that I will be getting data from. The second is data in those fields.
The fields list is like this:
|SchedGroup|GroupID|GroupName|IsValid|IsVM|VMName|TimerPeriod|CPUAllocMin|CPUAllocMax|
I converted this into a Selected.System.String named $Headercols:
header
------
SchedGroup
GroupID
GroupName
IsValid
IsVM
VMName
TimerPeriod
CPUAllocMin
CPUAllocMax
But I actually need it to be in columns, instead of rows. Because later on I want to add "records" / "rows" containing the data. I tried using converting to columns using add-member, but that needs a name and a value, but I don't know the value yet.
The example here only has a few rows, but this grows to 33 rows, so I'd rather fill them in a loop instead of addressing each one specifically.
Later on I will be receiving the data like this:
|SchedGroup|13567824|vm.4430541|1|1|BJLDC002|10416|0|-1|
|SchedGroup|12883482|vm.4316334|1|1|VCDJAGTEST|15260|0|-1|
|SchedGroup|13558449|vm.4429332|1|1|BJLADF02|1000|0|-1|
(Example has 3 rows, each one VM, but it can be many more rows with VMs.)
I know how to split the rows into separate values, but just as with the headers, I wouldn't know how to expand the object.
All examples I google are immediately adding name and value, I can't seem to get my head around how I should work with that.

But I actually need it to be in columns, instead of rows.
You're getting ahead of yourself - "columns" and "rows" relate to how we might format (or display) the data - but right now we need to import and parse the data somehow.
I would suggest trimming the leading and trailing |'s, at which point we can treat the data set as a CSV (with | instead of , as the separator - a PSV?):
# Read header list and split into individual strings
$headerString = Get-Content .\path\to\headers.txt |Select -First 1
$headers = $headerString.Split('|', [StringSplitOptions]::RemoveEmptyEntries)
# Read data set, remove the leading and trailing pipes
$dataSetRaw = (Get-Content .\path\to\data.txt) -replace '^\||\|$'
# Convert to structured objects
$data = $dataSetRaw |ConvertFrom-Csv -Delimiter '|' -Header $headers
ConvertFrom-Csv will create one object per record, at which point you'll find most formatting cmdlets do exactly what you expect:
PS ~> $data |Format-Table
SchedGroup GroupID GroupName IsValid IsVM VMName TimerPeriod CPUAllocMin CPUAllocMax
---------- ------- --------- ------- ---- ------ ----------- ----------- -----------
SchedGroup 13567824 vm.4430541 1 1 BJLDC002 10416 0 -1
SchedGroup 12883482 vm.4316334 1 1 VCDJAGTEST 15260 0 -1
SchedGroup 13558449 vm.4429332 1 1 BJLADF02 1000 0 -1

Continuing from my comment...
'|SchedGroup|GroupID|GroupName|IsValid|IsVM|VMName|TimerPeriod|CPUAllocMin|CPUAllocMax|' -replace '\|', ',' -replace '^,|,\s*$'
# Results
<#
SchedGroup,GroupID,GroupName,IsValid,IsVM,VMName,TimerPeriod,CPUAllocMin,CPUAllocMax
#>
Yet, you say your data is coming in this way...
SchedGroup,13567824,vm.4430541,1,1,BJLDC002,10416
SchedGroup,12883482,vm.4316334,1,1,VCDJAGTEST,15260
SchedGroup,13558449,vm.4429332,1,1,BJLADF02,1000
...then just import it and assign headers.
$Headers = #('|SchedGroup|GroupID|GroupName|IsValid|IsVM|VMName|TimerPeriod|CPUAllocMin|CPUAllocMax|' -replace '\|', ',' -replace '^,|,\s*$')
Import-Csv -Path 'SomeFilePath' -Header $Headers

Related

How to ignore double quotes which were added when comma was added in column value in pipe delimited csv file?

I have a pipe delimited csv file whose data are to be inserted into sql server.
Then I
Opened that file in excel and added comma in column values.
Then saved.
Here is the data in notepad++ after adding comma.
Then I bulk inserted that file into sql server.
BULK INSERT #temp1
FROM '\\as03\AppData\PipeDelimited.csv'
WITH (
FIRSTROW = 2
, FIELDTERMINATOR ='|'
, ROWTERMINATOR='\n'
, ERRORFILE ='\\as03\AppData\ErrorFiles\PipeDelimited.csv'
, CODEPAGE = '65001'
**strong text**, MAXERRORS = 99999999
)
But got the double quotes (") in first and last column values and also got two consecutive double quotes ("") where one double quote was already in file.
Here is the data inserted in sql server.
Is where some way to insert data in sql server by ignoring double quotes that were added by excel or notepad????
This appears to be a non-(PowerShell)issue as it actually works fine in PowerShell:
$Csv = #'
Name|Country|Address|Mobile
Tom|Nepal|Kathmandu,Nardevi|98456667365
Harry|India|Delhi"Patna,™ or R|9856524524
'# -Split '[\r?\n]'
$Csv |ConvertFrom-Csv -Delimiter '|' # Import-Csv .\PipeDelimited.csv -Delimiter '|'
Yields:
Name Country Address Mobile
---- ------- ------- ------
Tom Nepal Kathmandu,Nardevi 98456667365
Harry India Delhi"Patna,™ or R 9856524524
In other words, you might simply convert your PipeDelimited.csv to a more common CommaDelimited.csv with text indicators like:
Import-Csv .\PipeDelimited.csv -Delimiter '|' |Export-Csv .\CommaDelimited.csv
Your file was corrupted by editting it in Excel and saving it as a CSV. The best solution is to not use Excel to edit such files, but rather either use scripting or a text editor (depending on the complexity - just adding a comma to one field feels easiest in a text editor).
However; if we're saying the damage is done and you need a script to fix the issues caused you could run something like this; this reads in the data as plain text, applies a regex to remove the offending quotes, then spits the result out to a copy of the file (I've written to a copy rather than back to the original so you can rollback more easily if this wasn't what you wanted).
[string]$sourcePath = '.\PipeDelimited.csv'
[string]$outputPath = '.\PipeDelimitedCorrected.csv'
[System.Text.Encoding]$encoding = [System.Text.Encoding]::UTF8 # based on your code page
[string[]]$data = [System.IO.File]::ReadAllLines($sourcePath, $encoding)
$data = $data | ForEach-Object {((($_ -replace '^"', '') -replace '"$', '') -replace '""', '"')}
# replaces quotes at the start and end of the line with blanks,
# then replaces escaped double quotes with individual double quotes...
# it's a quick and dirty approach, but looking at your example should be sufficient
[System.IO.File]::WriteAllLines($outputPath, $data, $encoding)

Powershell - Parse duplicates in a list

I'm working on an issue with SCCM delivered App-V connection groups. Occasionally it delivers multiple (duplicate) connection groups to the client and the apps don't function correctly.
I'm running get-appvclientconnectiongroups in user context and where duplicates exist, exporting out Name, Groupid and Version id to a CSV.
I then import this using an elevated Powershell session (as I need admin rights to remove connection groups).
So CSV headers are
Name, GroupID, VersionID
The duplication lies in the Name header only
E.g.
Name, Group ID, Version ID
Adobe_Reader, 123, 456
Adobe_Reader, 456, 789
Adobe_Reader, 111, 555
Notepad, 333,222
Notepad, 111,444
Receiver, 444,777
Receiver, 123,999
What I would like to do is, for each duplicate name grab, the Group ID and Version ID to use in a remove-appvclientconnectiongroup. HOWEVER - I don't wish to do this for each entry - I want to stop when there is one left of each name (i.e when than name becomes unique in the list).
So in the end the list would be:
Adobe_Reader, 111, 555
Notepad, 111,444
Receiver, 123,999
And these are the ones we don't want to run throught the cmdlet
Any ideas? APologies if that makes no sense!
I've been playing around with arrays but not getting anywhere fast.
Assuming you have a CSV file already, you can do the following to return the last item in a group of identical names:
Import-Csv file.csv | Group-Object Name |
Foreach-Object { $_.Group[-1] }
Explanation:
Using Group-Object, you can group objects based on a property value. Here, grouping by property Name creates collection of items with properties Count,Name,Group. Name contains the values of the property you are grouping by. Count contains the number of matching values of that grouped property. Group contains the objects that had matching property values.
Since the Group property contains your objects, you can access the objects using the member access operator .. When piping to Foreach-Object, $_.Group will return the object groupings. Then you simply need to grab the last element [-1] of the collection.
If you have that information stored in a CSV file, you can do this to remove all but the last connection group:
Import-Csv -Path 'TheGroups.csv' | Group-Object Name | Where-Object { $_.Count -gt 1 } | Foreach-Object {
$last = $_.Count - 2
# remove all but the last connection group
$_.Group[0..$last] | Remove-AppvClientConnectionGroup
}
Thanks! I managed to get it working with the code below after much messing about. As there can be multiple instances of duplicates I pushed everything into an editable array which a removes line from as the groups are removed. It then checks how many duplicates are left for any given package and stops when there's one of each left
$data = import-csv $env:ALLUSERSPROFILE\AppvDuplciateGroups.csv
#Push the data into an ArrayList so it can be edited on the fly
$dataarray = [System.Collections.ArrayList]($data)
#Get the array size, number of duplicates and target array size
$arraysize = $dataarray.Count
$dupescount = $dataarray| group name
$arraytargetsize = $dupescount.count
$i = $arraysize -1
Function RemoveDuplicates(){
#select the relevant duplicate from the array based in index number (running bottom to top)
$lineX = $dataarray | select -Index $i
 #remove the duplicate
Remove-AppvClientConnectionGroup -GroupId $lineX.groupid -VersionId $lineX.VersionId
 #remove enrty from the array
$dataarray.RemoveAt($i)
 #check to see if that was the last entry for that particular connection group
$getcount = $dataarray | group-object name| Select-Object name, count | where name -eq $lineX.name
 #if there's still more that one entry for that package, run the function again on the next line up
If ($getcount.count -gt 1){
$i = $i -1
RemoveDuplicates}
 #if the array size is still larger than the calculated target array size, move up 2 lines in the array and run the function again
Else{
    If ($dataarray.count -gt $arraytargetsize){
        $i = $i -2
        RemoveDuplicates}
 #once the array meets the calculated target size, repair all connection groups and remove .csv file         
          
          Else{
                Repair-AppvClientConnectionGroup *
                Remove-Item -Path $env:ALLUSERSPROFILE\AppvDuplicateGroups.csv}
}
}
RemoveDuplicates ```

Selecting Distinct Items within Array using PowerShell and Linq

I have been banging my head on this problem for a few hours.
I have a multi-dimensional array and I need to select the unique items based on two "columns".
Is there an efficient .Net or otherwise way to do this and achieve the desired output?
The data looks something like:
ComputerName, IPAddress, MacAddress, FirstObserved
I would like to determine unique values based on MacAddress and ComputerName and keep the unique value based on the oldest FirstObserved date.
I have tried the PowerShell ways of doing this but it's horribly slow to say the least.
$data | Group-Object -Property ComputerName,MacAddress | ForEach-Object{$_.Group | Sort-Object -Property FirstObserved | Select-Object -First 1}
In a perfect world I would have a list of items no duplicates with the oldest entry based on FirstObserved date.
You can implement the grouping manually with a hashtable:
$FirstObserved = #{}
Import-Csv .\data.csv |ForEach-Object {
$key = $_.Computername,$_.MacAddress -join ','
$date = $_.FirstObserved -as [datetime]
# Check if we already have an entry for this Name + MAC combo
if($FirstObserved.Contains($key))
{
# Make sure the current date is older than what we have already
if($FirstObserved[$key] -gt $date)
{
$FirstObserved[$key] = $date
}
}
else
{
# First instance of the Name + MAC combo, save the date
$FirstObserved[$key] = $date
}
}
Now you can eaily look up the first date for a combination with:
$FirstObserved["Computer-01,11:22:33:44:55:66"]
If you want to export the list to another csv, all you need to do is turn the key-value pairs of the hashtable into objects:
$FirstObserved |Select-Object #{Name='Identity';Expression={$_.Key}},#{Name='FirstObserved';Expression={$_.Value}} |Export-Csv -Path output.csv

why is FORMAT not working with expandproperty

I am trying to check if the latest row in the table was added at the current time using the following script check:
$updateTime = "SELECT TOP 1 FORMAT([UPDATE_TIME], 'M/d/yyyy h:mm tt') FROM [dbo].[$Table] ORDER BY UPDATE_TIME DESC" |
Select -ExpandProperty UPDATE_TIME;
Before adding format, the expand property was working very well as intended getting me the latest UPDATE_TIME value
$updateTime = "SELECT TOP 1 [UPDATE_TIME] FROM [dbo].[$Table] ORDER BY UPDATE_TIME DESC" |
Select -ExpandProperty UPDATE_TIME;
However, when I added FORMAT to the query, it results in this error:
Select : Property "UPDATE_TIME" cannot be found.
... ble] ORDER BY UPDATE_TIME DESC" | Select -ExpandProperty UPDATE_TIME;
I need to use FORMAT because I want to ignore the seconds, or if there is away i can ignore a 1 second difference between the UPDATE_TIME value and Get-Date value, that would be even better.
$today = (Get-Date).ToString('M/d/yyyy h:mm tt')
if ($updateTime -eq $today) {
Write-Host "`r`n generated successfully at [ $updateTime ]"
} else {
Write-Host "`r`n NOT generated at [ $today ]"
}
With FORMAT but not expand property, i get this output:
Does your PowerShell console auto-execute strings as SQL queries? PowerShell doesn't normally do that.
Anyway, assuming that your PowerShell instance actually works that way, using the FORMAT() function (or other functions) changes the title of the result column, which in turn becomes the name of the respective property of the output object(s). Because of that the property UPDATE_TIME doesn't exist on those object and thus cannot be expanded.
You need to define a column name to fix that:
SELECT TOP 1 FORMAT(...) AS UPDATE_TIME ...

Powershell Grouping of csv import file

Hopefully I'm just missing something that is simple...
I have a csv file similar to this :
Employee ID, Entry Date, Product Code,Amount Due
0001,20/11/2017,A001,10
0001,20/11/2017,Q003,13
0001,20/11/2017,H001,8
0002,20/11/2017,P003,12
0002,20/11/2017,A001,7
and what I want as an output is similar to this :
0001;<some header text>;200171120
A001;10
Q003;13
H001;8
0002;<some header text>;200171120
P003;12
A001;7
So that each detail section is grouped by the Employee ID it relates to
I have tried piping the group-object ("Employee ID") after using an import-csv ... but I can't get it to work correctly
Any help much appreciated!
I think this does what you need:
$CSV | Group-Object 'Employee ID' | ForEach-Object {
$EntryDate = (Get-Date ($_.Group.'Entry Date' | Select -First 1) -F 'yyyyMMdd')
"{0};<some header text>;{1}" -f $_.Name, $EntryDate
ForEach ($Item in $_.Group) {
"{0},{1}" -f $Item.'Product Code',$Item.'Amount Due'
}
}
Explanation:
Uses Group-Object to group the results by Employee ID and then ForEach-Object to iterate through the collection.
Gets the Entry date from the first entry in the group, converts it to a date object with Get-Date and then formats it as year/month/day (this part assumes you don't care if there are other/differing dates in the collection).
Outputs the header string you wanted, using {0},{1} as placeholders for the variables passed via -f (there are multiple ways to do this but this seemed tidiest for your scenario).
Uses ForEach to iterate through the Group property, which contains the grouped items. Again using string replacement to output the product code and amount due fields in the format you desired.

Resources