Remove Empty Object in JSON Array using powershell - arrays

I have this JSON and I want to remove the empty object inside value array using powershell. Anyone can help?
{ "value" : [{},{},{"name": "a"}] }

# Parse the JSON into an object graph.
$obj = '{ "value" : [{},{},{"name": "a"}] }' | ConvertFrom-Json
# Filter out the empty objects, by counting the number of properties
# via the .psobject.Properties collection, available on any object in PowerShell.
# Note the need for #(...) to ensure that the result of the filtering remains
# an array.
$obj.Value = #($obj.Value | Where-Object { $_.psobject.Properties.Count -gt 0 })
# Reconvert to JSON.
$obj | ConvertTo-Json -Compress
The above yields:
{"value":[{"name":"a"}]}
See also:
ConvertFrom-Json
ConvertTo-Json
Where-Object

Related

Return the "Name" of an object that has a specific "Value" in PSObject

I have converted a JSON file to a PSObject using:
$json = Get-Content $filepath -Raw | ConvertFrom-Json
An example of what is now in the PSObject:
Value : Production
MemberType : NoteProperty
IsSettable : True
IsGettable : True
TypeNameOfValue : System.String
Name : Environment
IsInstance : True
I know you can get the Value of this by using:
$json.psobject.properties["Environment"].Value
This will return "Production"
Question:
Is there a way to get the Name returned based on the value like how I am able to get the Value returned based on the name as shown above?
I.e. How can i get "Environment" returned?
For background I am writing a script that will loop through all the values and if the values are empty then i need to print the Names.
With Theo's help you've already found the solution, but let me spell it out, with some background information:
# Read a sample object from JSON text.
$fromJson = #'
{
"Foo": "Bar",
"Environment": "Production",
"Other": "Stuff"
}
'# | ConvertFrom-Json
# Find the name of the property/ies whose value is 'Production'
# -> 'Environment'
$fromJson.psobject.Properties.Where({ $_.Value -eq 'Production' }).Name
Note:
The above relies on the intrinsic psobject property that PowerShell exposes on all objects, which is a rich source of reflection, such as on a given object's properties via .psobject.Properties, as well as on the intrinsic .Where() method for filtering (a better-performing, feature-richer alternative to the Where-Object cmdlet).
The code only works for top-level properties of the object(s) parsed from JSON. If you need to search an entire object graph, i.e. nested objects for values at any level of the hierarchy, more effort is needed.
The following example assumes that function Get-PropertyPathByValue has already been defined (see below):
# Read a *nested* sample object from JSON text.
$fromJson = #'
{
"Foo": "Bar",
"Nested": {
"More": "Stuff",
"Environment": "Production"
},
"Other": "Stuff"
}
'# | ConvertFrom-Json
# Find the value 'Production' anywhere in the object hierarchy,
# and return the matching property's name *path* (dot notation).
# -> 'Nested.Environment'
$fromJson | Get-PropertyPathByValue -Value 'Production'
Get-PropertyPathByValue source code - note the limitation on what types of values can be searched for:
# Finds a *simple* value in the object graphs given and returns the path of
# matching properties in dot notation (e.g., 'foo.bar`).
# Note: *Simple* means a *single* (non-collection) value that is either:
# * a string or
# * an instance of a .NET primitive or similar type (numbers, dates).
function Get-PropertyPathByValue {
param([Parameter(ValueFromPipeline, Mandatory)] [AllowNull()] [object] $InputObject, [Parameter(Mandatory)] [AllowNull()] $Value, [string] $NamePath)
process {
if ($null -eq $InputObject -or $InputObject.GetType().IsPrimitive -or $InputObject.GetType() -in [string], [datetime], [datetimeoffset], [decimal], [bigint]) {
# A null-like value or a primitive / quasi-primitive type -> output.
if ($Value -eq $InputObject) { return $NamePath }
}
elseif ($InputObject -is [System.Collections.IEnumerable] -and $InputObject -isnot [System.Collections.IDictionary]) {
# A collection of sorts (other than a string or dictionary (hash table)),
# recurse on its elements.
$i = 0
foreach ($o in $InputObject) { Get-PropertyPathByValue $o $Value ($NamePath + '[' + $i++ + ']') }
}
else {
# A non-quasi-primitive scalar object or a dictionary:
# enumerate its properties / entries.
$props = if ($InputObject -is [System.Collections.IDictionary]) { $InputObject.GetEnumerator() } else { $InputObject.psobject.properties }
$sep = '.' * ($NamePath -ne '')
foreach ($p in $props) {
Get-PropertyPathByValue $p.Value $Value ($NamePath + $sep + $p.Name)
}
}
}
}

Powershell nested JSON to csv conversion

I have a rather peculiar nested JSON where in some instances a key - value pair occurs as normal, but in others the type of the key appears in a further nesting.
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
In the example you can see metadata's fields are straightforward key-value pairs, but underneath recordContent, we have positionDate which is a straightforward key-value but "account":{"string":"G32"} and "strike":{"double":4.4} are not.
I'd like to ditch the type information and arrive at a CSV structure as follows:
systemId, legalEntity, positionDate, account,seg,strike
da1895, A0, 2019-04-08 00:00:00.0,G32, S, 4.4
4536d, 5G, 2019-04-08 00:00:00.0,G81, S, 5.0
Any ideas on how to convert such a structure to CSV using Powershell?
Here's what I tried:
$TemplateParametersFile = "c:\data\output.json"
$JsonParameters = Get-Content $TemplateParametersFile | ConvertFrom-Json
$metadatafields = $JsonParameters.metadata[0].PSObject.Properties.Name
$recordcontentfields = $JsonParameters.recordContent[0].PsObject.Properties.Name
$oData = New-Object PSObject
$metadatafields |
ForEach {
Add-Member -InputObject $oData -NotePropertyName ($_) -NotePropertyValue $JsonParameters.metadata.($_)
}
$recordcontentfields |
ForEach {
Add-Member -InputObject $oData -NotePropertyName ($_) -NotePropertyValue $JsonParameters.recordContent.($_)
}
This gave me:
$oData
systemId : {da1895, 45364d}
legalEntity : {A0, 5G}
positionDate : {2019-04-08 00:00:00.0, 2019-04-08 00:00:00.0}
account : {#{string=G32}, #{string=G81}}
seg : {#{string=S}, #{string=S}}
strike : {#{double=4.4}, #{double=5.0}}
I'm a bit stuck now and the above doesn't convert to csv.
Note that other than metadata and recordContent, I've not hardcoded any fieldnames and I'd like to maintain that flexibility in case the JSON structure changes.
Thanks
I suggest collecting the property-name-value pairs iteratively in an ordered hashtable ([ordered] #{}), which can then be cast to [pscustomobject] to convert it to a custom object.
No property names are hard-coded in the following solution, but the object-graph structure is assumed to follow the pattern in your sample JSON, which is limited to one level of nesting - if you need to process arbitrarily nested objects, this answer may be a starting point.
Reflection (discovery of the property names and values) is performed via the intrinsic .psobject property that PowerShell makes available on all objects.
# Parse sample JSON into an array of [pscustomobject] graphs.
$fromJson = ConvertFrom-Json #'
[
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
,
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
]
'#
# Initialize an aux. ordered hashtable to collect the property-name-value
# pairs in.
$oht = [ordered] #{}
$fromJson | ForEach-Object {
$oht.Clear()
# Loop over top-level properties.
foreach ($topLevelProp in $_.psobject.Properties) {
# Loop over second-level properties.
foreach ($prop in $topLevelProp.Value.psobject.Properties) {
if ($prop.Value -is [System.Management.Automation.PSCustomObject]) {
# A nested value: Use the value of the (presumed to be one-and-only)
# property of the object stored in the value.
$oht[$prop.Name] = $prop.Value.psobject.Properties.Value
}
else {
# A non-nested value: use as-is.
$oht[$prop.Name] = $prop.Value
}
}
}
# Construct and output a [pscustomobject] from the aux. ordered hashtble.
[pscustomobject] $oht
} |
ConvertTo-Csv # Replace this with Export-Csv to export to a file.
The above yields:
"systemId","legalEntity","positionDate","account","seg","strike"
"da1895","A0","2019-04-08 00:00:00.0","G32","S","4.4"
"45364d","5G","2019-04-08 00:00:00.0","G81","S","5"
A few years ago, I wrote a reusable Flatten-Object function for this.
The only difference is that it combines the (sub)property names with the parent property names as they might not be unique:
$JsonParameters |Flatten-Object |Format-Table
metadata.systemId metadata.legalEntity recordContent.positionDate recordContent.account.string recordContent.seg.string recordContent.strike.double
----------------- -------------------- -------------------------- ---------------------------- ------------------------ ---------------------------
da1895 A0 2019-04-08 00:00:00.0 G32 S 4.4
45364d 5G 2019-04-08 00:00:00.0 G81 S 5
Try this:
$data = ConvertFrom-Json #"
[
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}},
{"metadata":{"systemId":"45364d","legalEntity":"5G"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G81"},"seg":{"string":"S"},"strike":{"double":5.0}}}
]
"#
$data | Select-Object -Property #{l="systemId"; e={$_.metadata.systemId}}, #{l="legalEntity"; e={$_.metadata.legalEntity}},
#{l="positionDate"; e={$_.recordContent.positionDate}}, #{l="account"; e={$_.recordContent.account.string}},
#{l="seg"; e={$_.recordContent.seg.string}}, #{l="strike"; e={$_.recordContent.strike.double}} | Export-Csv
This should work with any nested psobject.
$json = #'
{"metadata":{"systemId":"da1895","legalEntity":"A0"},"recordContent":{"positionDate":"2019-04-08 00:00:00.0","account":{"string":"G32"},"seg":{"string":"S"},"strike":{"double":4.4}}}
'#
$obj = ConvertFrom-Json $json
$obj.recordContent | gm -MemberType NoteProperty | % {
$prop = $_.name
if ($obj.recordContent.$prop.GetType().name -eq 'pscustomobject') {
$obj.recordContent.$prop = $obj.recordContent.$prop.psobject.Members | where membertype -eq noteproperty | select -ExpandProperty value
}
$obj.metadata | add-member -MemberType NoteProperty -Name $prop -Value $obj.recordContent.$prop
}
$newobj = $obj.metadata
$newobj

Compare Two Objects Properties and Obtain Missing Items

I'm attempting to compare two object properties called "name" and that contain the same type of data, in this case server host names. The objects are not identical AKA do not have the same properties. In short, I'm attempting to compare two lists of server names and determine where (which object) they are missing from.
I'm looking to find the items (server host names) that are missing from each object. When something is found I'm hoping to obtain all related properties for the item in the given object that it was found in. I can do the compare-object successfully, but don't know how to get the results I'm looking for.
I'm thinking two new objects could be created for each, that list the items that were not found in the other object maybe? Or do I somehow reference the previous objects with the output from compare-object and produce some formatted output?
This code currently produces a blank file.
Data Format:
$SNObject:
name,ip,class
server-place.com,10.10.10.10,windows server
$QRObject:
name,date,ip,general,device type
server-place1.com,11.11.11.11,random info,linux server
Code Example:
$compare = compare-object $SNObject $QRObject -property Name |
foreach {
if ($_.sideindicator -eq '<=')
{$_.sideindicator = $PathToQRReport }
if ($_.sideindicator -eq '=>')
{$_.sideindicator = $PathToSNReport}
}
$compare |
select #{l='Value';e={$_.InputObject}},#{l='File';e={$_.SideIndicator}} |
Out-File -FilePath C:\Temp\MissingOutputs1.txt
Ahh... just thought of an alternative that may give you exactly what you're looking for but in a slightly different way:
## Join both arrays into single array and then group it on the property name that has a shared value, 'name'
$all = #()
$group = #($SNObject + $QRObject)
$group | Group-Object -Property name | % {
## Create a custom object that contains all possible properties plus a directionality indicator ('source')
$n = New-Object PSObject -Property #{
'name' = ''
'date' = ''
'ip' = ''
'general' = ''
'platform' = ''
'source' = ''
}
if ($_.Count -eq 1) {
## Loop through the grouped results and determine their source and write properties based off of their source
foreach ($i in $_.Group) {
if (#($i | Get-Member -MemberType NoteProperty | Select-Object -ExpandProperty name) -contains 'date' ) {
## This value came from $QRObject which apparently is the only dataset that contains a date property
$n.source = 'QRObject'
$n.date = $i.date
$n.general = $i.general
$n.platform = $i.'device type'
} else {
## This object does not contain the 'date' property, therefore it came from $SNObject
$n.source = 'SNObject'
$n.platform = $i.class
}
## write out common properties
$n.name = $i.name
$n.ip = $i.ip
## add the custom PSObject back to a master array with all formatted properties
$all += $n
}
}
}
$all | out-whereever-you-want

Select-String bug splitting up lines [duplicate]

I'm using Powershell to set up IIS bindings on a web server, and having a problem with the following code:
$serverIps = gwmi Win32_NetworkAdapterConfiguration
| Where { $_.IPAddress }
| Select -Expand IPAddress
| Where { $_ -like '*.*.*.*' }
| Sort
if ($serverIps.length -le 1) {
Write-Host "You need at least 2 IP addresses for this to work!"
exit
}
$primaryIp = $serverIps[0]
$secondaryIp = $serverIps[1]
If there's 2+ IPs on the server, fine - Powershell returns an array, and I can query the array length and extract the first and second addresses just fine.
Problem is - if there's only one IP, Powershell doesn't return a one-element array, it returns the IP address (as a string, like "192.168.0.100") - the string has a .length property, it's greater than 1, so the test passes, and I end up with the first two characters in the string, instead of the first two IP addresses in the collection.
How can I either force Powershell to return a one-element collection, or alternatively determine whether the returned "thing" is an object rather than a collection?
Define the variable as an array in one of two ways...
Wrap your piped commands in parentheses with an # at the beginning:
$serverIps = #(gwmi Win32_NetworkAdapterConfiguration
| Where { $_.IPAddress }
| Select -Expand IPAddress
| Where { $_ -like '*.*.*.*' }
| Sort)
Specify the data type of the variable as an array:
[array]$serverIps = gwmi Win32_NetworkAdapterConfiguration
| Where { $_.IPAddress }
| Select -Expand IPAddress
| Where { $_ -like '*.*.*.*' }
| Sort
Or, check the data type of the variable...
IF ($ServerIps -isnot [array])
{ <error message> }
ELSE
{ <proceed> }
Force the result to an Array so you could have a Count property. Single objects (scalar) do not have a Count property. Strings have a length property so you might get false results, use the Count property:
if (#($serverIps).Count -le 1)...
By the way, instead of using a wildcard that can also match strings, use the -as operator:
[array]$serverIps = gwmi Win32_NetworkAdapterConfiguration -filter "IPEnabled=TRUE" | Select-Object -ExpandProperty IPAddress | Where-Object {($_ -as [ipaddress]).AddressFamily -eq 'InterNetwork'}
You can either add a comma(,) before return list like return ,$list or cast it [Array] or [YourType[]] at where you tend to use the list.
If you declare the variable as an array ahead of time, you can add elements to it - even if it is just one...
This should work...
$serverIps = #()
gwmi Win32_NetworkAdapterConfiguration
| Where { $_.IPAddress }
| Select -Expand IPAddress
| Where { $_ -like '*.*.*.*' }
| Sort | ForEach-Object{$serverIps += $_}
You can use Measure-Object to get the actual object count, without resorting to an object's Count property.
$serverIps = gwmi Win32_NetworkAdapterConfiguration
| Where { $_.IPAddress }
| Select -Expand IPAddress
| Where { $_ -like '*.*.*.*' }
| Sort
if (($serverIps | Measure).Count -le 1) {
Write-Host "You need at least 2 IP addresses for this to work!"
exit
}
Return as a referenced object, so it never converted while passing.
return #{ Value = #("single data") }
I had this problem passing an array to an Azure deployment template. If there was one object, PowerShell "converted" it to a string. In the example below, $a is returned from a function that gets VM objected according to the value of a tag. I pass the $a to the New-AzureRmResourceGroupDeployment cmdlet by wrapping it in #(). Like so:
$TemplateParameterObject=#{
VMObject=#($a)
}
New-AzureRmResourceGroupDeployment -ResourceGroupName $RG -Name "TestVmByRole" -Mode Incremental -DeploymentDebugLogLevel All -TemplateFile $templatePath -TemplateParameterObject $TemplateParameterObject -verbose
VMObject is one of the template's parameters.
Might not be the most technical / robust way to do it, but it's enough for Azure.
Update
Well the above did work. I've tried all the above and some, but the only way I have managed to pass $vmObject as an array, compatible with the deployment template, with one element is as follows (I expect MS have been playing again (this was a report and fixed bug in 2015)):
[void][System.Reflection.Assembly]::LoadWithPartialName("System.Web.Extensions")
foreach($vmObject in $vmObjects)
{
#$vmTemplateObject = $vmObject
$asJson = (ConvertTo-Json -InputObject $vmObject -Depth 10 -Verbose) #-replace '\s',''
$DeserializedJson = (New-Object -TypeName System.Web.Script.Serialization.JavaScriptSerializer -Property #{MaxJsonLength=67108864}).DeserializeObject($asJson)
}
$vmObjects is the output of Get-AzureRmVM.
I pass $DeserializedJson to the deployment template' parameter (of type array).
For reference, the lovely error New-AzureRmResourceGroupDeployment throws is
"The template output '{output_name}' is not valid: The language expression property 'Microsoft.WindowsAzure.ResourceStack.Frontdoor.Expression.Expressions.JTokenExpression'
can't be evaluated.."
There is a way to deal with your situation. Leave most of you code as-is, just change the way to deal with the $serverIps object. This code can deal with $null, only one item, and many items.
$serverIps = gwmi Win32_NetworkAdapterConfiguration
| Where { $_.IPAddress }
| Select -Expand IPAddress
| Where { $_ -like '*.*.*.*' }
| Sort
# Always use ".Count" instead of ".Length".
# This works on $null, only one item, or many items.
if ($serverIps.Count -le 1) {
Write-Host "You need at least 2 IP addresses for this to work!"
exit
}
# Always use foreach on a array-possible object, so that
# you don't have deal with this issue anymore.
$serverIps | foreach {
# The $serverIps could be $null. Even $null can loop once.
# So we need to skip the $null condition.
if ($_ -ne $null) {
# Get the index of the array.
# The #($serverIps) make sure it must be an array.
$idx = #($serverIps).IndexOf($item)
if ($idx -eq 0) { $primaryIp = $_ }
if ($idx -eq 1) { $secondaryIp = $_ }
}
}
In PowerShell Core, there is a .Count property exists on every objects. In Windows PowerShell, there are "almost" every object has an .Count property.

Powershell: Reformat JSON nested Arrays to Array of Objects

I'm trying to modify a badly formated JSON Textfile, that currently looks like this:
[
["val1", "val2", "val3", "val4"],
["val5", "val6", "val7", "val8"],
["val9", "val10", "val11", "val12"]
]
and I have an other array containing field names
["title1", "title2", "title3", "title4"]
I want to output a final.json textfile looking like this:
[
{"title1": "val1", "title2": "val2", "title3": "val3", "title4": "val4"},
{"title1": "val5", "title2": "val6", "title3": "val7", "title4": "val8"},
{"title1": "val9", "title2": "val10", "title3": "val11", "title4": "val12"}
]
I guess the best way would be to take each row, split by , and then adding them back together foreach-ing over the title names, but I'm not quite sure on how to do this in PowerShell.
Since you're dealing with structured data here, I think the best way is to parse the JSON, and work with the resulting objects. Create the objects you want, then convert back to JSON:
$j1 = #'
[
["val1", "val2", "val3", "val4"],
["val5", "val6", "val7", "val8"],
["val9", "val10", "val11", "val12"]
]
'#
$j2 = #'
["title1", "title2", "title3", "title4"]
'#
$a1 = $j1 | ConvertFrom-Json
$a2 = $j2 | ConvertFrom-Json
0..($a1.Count-1) | ForEach-Object {
$i = $_
$props = #{}
0..($a2.Count-1) | ForEach-Object {
$props[$a2[$_]] = $a1[$i][$_]
}
New-Object PSOBject -Property $props
} | ConvertTo-Json
ConvertTo-Json and ConvertFrom-Json are the cmdlets you need to serialize/deserialize the JSON. Then you just work with the objects.
In this case I'm going through each top level array in $a1 and creating a hashtable that contains the properties you want. Then I'm creating a PSObject with those properties. That gets returned from the ForEach-Object cmdlet (the result is an array of those objects) which then gets piped directly into ConvertTo-Json to give you the output needed.
I think a better approach is to read in the first JSON format via ConvertFrom-Json, then take that array of arrays and for each row, create a PSCustomObject from a hashtable e.g. [PSCustomObject]#{title1=$arr[$row][0]; title2=$arr[$row][1];...}. Once you then have that array of PSCustomObject, convert that back to JSON with ConvertTo-Json.

Resources