Working with a nested arrays like this:
[
{
"value1": "Data1-0",
"value2": "Data2-0",
"nArray": [
{"nValue1": "nData1-0a","nValue2": "nData2-0a"},
{"nValue1": "nData1-0b","nValue2": "nData2-0a"},
{"nValue1": "nData1-0c","nValue2": "nData2-0a"}
],
"value3": "Data3-0"
},
{
"value1": "Data1-1",
"value2": "Data2-1",
"nArray": [
{"nValue1": "nData1-1a","nValue2": "nData2-1a"},
{"nValue1": "nData1-1b","nValue2": "nData2-1a"}
],
"value3": "Data3-1"
}
]
Desired output is CSV format like this:
Value1,Value2,nArrayCount
Data1-0,Data2-0,Data3-0,3
Data1-1,Data2-1,Data3-1,2
I was able to get the nested values but that produces multiple rows for each nArray value with this:
[.[] | [.value1,.value2,.value3] + (.nArray[]? | [.nValue1]) ] | .[] | #csv
All I need is a count.
If the data is like the one in your post (no double quotes or comma in Values), then :
#!/usr/bin/env bash
jq -r '["Value1", "Value2", "Value3", "nArrayCount"],
(.[] | [.value1, .value2, .value3, (.nArray|length)])
| join(",")' input.json
Close enough. Use length to get the length of an array.
$ jq -r '
["Value1", "Value2", "count"],
(.[] | [.value1, .value2, (.nArray|length)])
| #csv' input.json
"Value1","Value2","count"
"Data1-0","Data2-0",3
"Data1-1","Data2-1",2
$ jq -r '
["Value1", "Value2", "count"],
(.[] | [.value1,.value2] + [.nArray|length])
| #csv' input.json
"Value1","Value2","count"
"Data1-0","Data2-0",3
"Data1-1","Data2-1",2
Related
Given the example JSON below:
{
"account_number": [
"123456"
],
"account_name": [
"name"
],
"account_id": [
654321
],
"username": [
"demo"
]
}
I'd like to get:
{
"account_number": "123456",
"account_name": "name",
"account_id": 654321,
"username": "demo"
}
Currently, I'm brute forcing it with | sed 's/\[//g' | sed 's/\]//g' | jq '.' ... but of course, that's ugly and causes issues if any of the values contain [ or ].
I've been unsuccessful with jq's flatten and other loops and mapping techniques like | jq -s '{Item:.[]} | .Item |add' to try and flatten the single-item arrays. Ideally, it would work where it would flatten arrays [...] to flat elements/objects {...}. Either way something better than replacing all occurrences of square brackets.
Short and sweet:
map_values(first)
Use with_entries, changing each value to the first element of itself:
jq 'with_entries(.value |= .[0])' file.json
I intend to extract a csv with a row for each sub array item.
Given a json array with a sub array. e.g. like this one:
[
{
"foo": 108,
"bar": ["a","b"]
},
{
"foo": 201,
"bar": ["c","d"]
}
]
It is possible to fetch the data by utilizing an intermediate object.
.[] | { "y": .foo, "x": .bar[] }| [.y,.x] | #csv
https://jqplay.org/s/922RlkbFNA
But I'd like to express it in a less elaborate form.
However the following does not work :( :
.[] | [ (.foo, .bar[]) ] | #csv
PS: I struggle to find a fitting headline
In three lines:
.[]
| [.foo] + (.bar[]|[.])
| #csv
or maybe less obscurely:
.[]
| .bar[] as $bar
| [.foo, $bar]
| #csv
Is it possible to filter an entire array of items in JQ in only one pass? Compare the following code, which runs jq over and over:
{
"foofoo": {
"barbar": [
{
"foo": "aaaa",
"bar": 0000
},
{
"foo": "bbbb",
"bar": 1111
},
{
"foo": "cccc",
"bar": 2222
}
]
}
}
bash array:
array=("1111" "2222")
my code is working but not very efficient and uses a lot of resources considering the array size in reality:
for k in "${array[#]}"; do
jq --argjson k "$k" '.foofoo.barbar |= map(select(.bar != $k))' json.json | sponge json.json
done
It keeps looping through the array, removing the unneeded entries and storing same file again by using sponge.
any ideas how to achieve a similar behavior with a lighter code?
Desired output:
{
"foofoo": {
"barbar": [
{
"foo": "aaaa",
"bar": 0
}
]
}
}
To improve the performance significantly use the following jq approach (without any shell loops):
arr=("1111" "2222")
jq '($p | split(" ") | map(tonumber)) as $exclude
| .foofoo.barbar
|= map(select(.bar as $b
| any($exclude[]; . == $b) | not))' \
--arg p "${arr[*]}" file.json | sponge file.json
The output:
{
"foofoo": {
"barbar": [
{
"foo": "aaaa",
"bar": 0
}
]
}
}
I'm positive there are better ways to do this: I really just throw stuff at jq until something sticks to the wall ...
# 1. in the shell, construct a JSON object string from the array => {"bbbb":1,"cccc":1}
printf -v jsonobj '{%s}' "$(printf '"%q":1\n' "${array[#]}" | paste -sd,)"
# 2. use that to test for non-membership in the jq select function
jq --argjson o "$jsonobj" '.foofoo.barbar |= map(select((.bar|in($o)) == false))' json.json
outputs
{
"foofoo": {
"barbar": [
{
"foo": "0000",
"bar": "aaaa"
}
]
}
}
You don't actually show your desired output, so I assume this is what you want.
Constructing a dictionary object opens the door to an efficient solution. If your jq has INDEX/2, you could use the following invocation:
jq --arg p "${arr[*]}" '
INDEX($p | split(" ")[]; .) as $dict
| .foofoo.barbar
|= map(select($dict[.bar|tostring] | not))'
If your jq does not have INDEX/2, then now would be an excellent time to upgrade; otherwise, you could snarf its def by googling:
jq "def INDEX"
I have a dictionary which looks like:
cat dictionary.json
[
{
"key": "key01",
"value": "value01"
},
{
"key": "key02",
"value": "value02"
},
{
"key": "key03",
"value": "value03",
"extraProperty": {
"foo": "bar"
}
},
{
"key": "key04",
"value": "value04"
}
]
Then, I have an array which is:
echo $array
key01 key02 key03
Expected output:
value01 value02 value03
I have some trouble to make jq using an array which is not json format.
I tried various solutions that I found, but none of them worked.
This post jq - How to select objects based on a 'whitelist' of property values seems to solve a similar problem but it doesn't work with my input:
echo $array | jq --argfile whitelist dictionary.json 'select(any(.key== $whitelist[]; .value))'
parse error: Invalid numeric literal at line 1, column 6
I also tried to use
jq -n --arg array $array --argfile whitelist dico.json 'select(any(.key== $whitelist[]; .valuee))'
jq: error: key02/0 is not defined at <top-level>, line 1:
key02
jq: 1 compile error
Thanks!
Here
jq -r --arg array "$array" \
'from_entries | .[($array | split(" "))[]]' \
dictionary.json
Output
value01
value02
value03
See man jq for further information.
Using INDEX/2, which constructs a dictionary:
echo 'key01 key02 key03' |
jq -Rr --argfile dict dictionary.json '
INDEX($dict[]; .key) as $d
| split(" ") | map( $d[.]|.value )
| join(" ")'
yields:
value01 value02 value03
If your jq does not have INDEX, then now would be an excellent time to upgrade to jq 1.6; alternatively, you can simply snarf its def by googling: jq "def INDEX"
Struggling with formatting of data in jq. I have 2 issues.
Need to take the last array .rental_methods and concatenate them into 1 line, colon separated.
#csv doesn't seem to work with my query. I get the error string ("5343") cannot be csv-formatted, only array
jq command is this (without the | #csv)
jq --arg LOC "$LOC" '.last_updated as $lu | .data[]|.[]| $lu, .station_id, .name, .region_id, .address, .rental_methods[]'
JSON:
{
"last_updated": 1539122087,
"ttl": 60,
"data": {
"stations": [{
"station_id": "5343",
"name": "Lot",
"region_id": "461",
"address": "Austin",
"rental_methods": [
"KEY",
"APPLEPAY",
"ANDROIDPAY",
"TRANSITCARD",
"ACCOUNTNUMBER",
"PHONE"
]
}
]
}
}
I'd like the output to end up as:
1539122087,5343,Lot,461,Austin,KEY:APPLEPAY:ANDROIDPAY:TRANSITCARD:ACCOUNTNUMBER:PHONE:,
Using #csv:
jq -r '.last_updated as $lu
| .data[][]
| [$lu, .station_id, .name, .region_id, .address, (.rental_methods | join(":")) ]
| #csv'
What you were probably missing with #csv before was an array constructor around the list of things you wanted in the CSV record.
You could repair your jq filter as follows:
.last_updated as $lu
| .data[][]
| [$lu, .station_id, .name, .region_id, .address,
(.rental_methods | join(":"))]
| #csv
With your JSON, this would produce:
1539122087,"5343","Lot","461","Austin","KEY:APPLEPAY:ANDROIDPAY:TRANSITCARD:ACCOUNTNUMBER:PHONE"
... which is not quite what you've said you want. Changing the last line to:
map(tostring) | join(",")
results in:
1539122087,5343,Lot,461,Austin,KEY:APPLEPAY:ANDROIDPAY:TRANSITCARD:ACCOUNTNUMBER:PHONE
This is exactly what you've indicated you want except for the terminating punctuation, which you can easily add (e.g. by appending + "," to the program above) if so desired.