I have three operations with jq to get the right result. How can I do it within one command?
Here is a fragment from the source JSON file
[
{
"Header": {
"Tenant": "tenant-1",
"Rcode": 200
},
"Body": {
"values": [
{
"id": "0b0b-0c0c",
"name": "NumberOfSearchResults"
},
{
"id": "aaaa0001-0a0a",
"name": "LoadTest"
}
]
}
},
{
"Header": {
"Tenant": "tenant-2",
"Rcode": 200
},
"Body": {
"values": []
}
},
{
"Header": {
"Tenant": "tenant-3",
"Rcode": 200
},
"Body": {
"values": [
{
"id": "cccca0003-0b0b",
"name": "LoadTest"
}
]
}
},
{
"Header": {
"Tenant": "tenant-4",
"Rcode": 200
},
"Body": {
"values": [
{
"id": "0f0g-0e0a",
"name": "NumberOfSearchResults"
}
]
}
}
]
I apply two filters and create two intermediate JSON files. First I create the list of all tenants
jq -r '[.[].Header.Tenant]' source.json >all-tenants.json
And then I select to create an array of all tenants not having a particular key present in the Body.values[] array:
jq -r '[.[] | select (all(.Body.values[]; .name !="LoadTest") ) ] | [.[].Header.Tenant]' source.json >filter1.json
Results - all-tenants.json
["tenant-1",
"tenant-2",
"tenant-3",
"tenant-4"
]
filter1.json
["tenant-2",
"tenant-4"
]
And then I substruct filter1.json from all-tenants.json to get the difference:
jq -r -n --argfile filter filter1.json --argfile alltenants all-tenants.json '$alltenants - $filter|.[]'
Result:
tenant-1
tenant-3
Tenant names - values for the "Tenant" key are unique and each of them occurs only once in the source.json file.
Just to clarify - I understand that I can have a select condition(s) that would give me the same resut as subtracting two arrays.
What I want to understand - how can I assign and use these two arrays into vars directly in a single command not involving the intermediate files?
Thanks
Use your filters to fill in the values of a new object and use the keys to refer to the arrays.
jq -r '{
"all-tenants": [.[].Header.Tenant],
"filter1": [.[]|select (all(.Body.values[]; .name !="LoadTest"))]|[.[].Header.Tenant]
} | .["all-tenants"] - .filter1 | .[]'
Note: .["all-tenants"] is required by the special character "-" in that key. See the entry under Object Identifier-Index in the manual.
how can I assign and use these two arrays into vars directly in a single command not involving the intermediate files?
Simply store the intermediate arrays as jq "$-variables":
[.[].Header.Tenant] as $x
| ([.[] | select (all(.Body.values[]; .name !="LoadTest") ) ] | [.[].Header.Tenant]) as $y
| $x - $y
If you want to itemize the contents of $x - $y, then simply add a final .[] to the pipeline.
Related
Example input:
{
"firstName": "Jam",
"Product": [
{
"productId": "5e09ad38986b7c30f339c5c0"
},
{
"productId": "5e09407b986b7c30f339c18e"
},
{
"productId": "5e094c2a986b7c30f339c1d2"
}
]
}
Expected output:
Jam,5e09ad38986b7c30f339c5c0
Jam,5e09407b986b7c30f339c18e
Jam,5e094c2a986b7c30f339c1d2
Current command is producing the output but on the same row comma seperated:
jq -rc '.firstName,.Product[0] .productId'
To generate a report in CSV format you need to put column values into an array and pass it to #csv filter.
$ jq -r '[.firstName] + (.Product[] | [.productId]) | #csv' file
"Jam","5e09ad38986b7c30f339c5c0"
"Jam","5e09407b986b7c30f339c18e"
"Jam","5e094c2a986b7c30f339c1d2"
Is there a way to compare two json files in jq? Specifically, I'd like to be able to remove objects from one json file if they occur in another json file. Basically, subtract one file from another. It would be a bonus if I could generalize this so that I could define the equality criteria for the objects, but this is not strictly necessary, it can be based strictly on the objects being identical.
So the more general case would look like this. Let's say I have a file that looks like this:
[
{
"name": "Cynthia",
"surname": "Craig",
"isActive": true,
"balance": "$2,426.88"
},
{
"name": "Elise",
"surname": "Long",
"isActive": false,
"balance": "$1,892.72"
},
{
"name": "Hyde",
"surname": "Adkins",
"isActive": true,
"balance": "$1,769.34"
},
{
"name": "Matthews",
"surname": "Jefferson",
"isActive": true,
"balance": "$1,991.42"
},
{
"name": "Kris",
"surname": "Norris",
"isActive": false,
"balance": "$2,137.11"
}
]
And I have a second file that looks like this:
[
{
"name": "Cynthia",
"surname": "Craig"
},
{
"name": "Kris",
"surname": "Norris"
}
]
I'd like to remove any objects from the first file where the name and surname fields match an object of the second file, so that the results should look like this:
[
{
"name": "Elise",
"surname": "Long",
"isActive": false,
"balance": "$1,892.72"
},
{
"name": "Hyde",
"surname": "Adkins",
"isActive": true,
"balance": "$1,769.34"
},
{
"name": "Matthews",
"surname": "Jefferson",
"isActive": true,
"balance": "$1,991.42"
}
]
The following solution is intended to be generic, efficient and as simple as possible subject to the first two objectives.
Genericity
For genericity, let us suppose that $one and $two are two arrays of
JSON entities, and that we wish to find those items, $x, in $one
such that ($x|filter) does not appear in map($two | filter), where filter is an arbitrary filter. (In the present instance, it is {surname, name}.)
The solution uses INDEX/1, which was added to jq after the official 1.5 release, so we begin by reproducing its definition:
def INDEX(stream; idx_expr):
reduce stream as $row ({};
.[$row|idx_expr|
if type != "string" then tojson
else .
end] |= $row);
def INDEX(idx_expr): INDEX(.[]; idx_expr);
Efficiency
For efficiency, we will need to use a JSON object as a dictionary;
since keys must be strings, we will need to ensure that when converting an object
to a string, the objects are normalized. For this, we define normalize as follows:
# Normalize the input with respect to the order of keys in objects
def normalize:
. as $in
| if type == "object" then reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | normalize) } )
elif type == "array" then map( normalize )
else .
end;
To construct the dictionary, we simply apply (normalize|tojson):
def todict(filter):
INDEX(filter| normalize | tojson);
The solution
The solution is now quite simple:
# select those items from the input stream for which
# (normalize|tojson) is NOT in dict:
def MINUS(filter; $dict):
select( $dict[filter | normalize | tojson] | not);
def difference($one; $two; filter):
($two | todict(filter)) as $dict
| $one[] | MINUS( filter; $dict );
difference( $one; $two; {surname, name} )
Invocation
$ jq -n --argfile one one.json --argfile two two.json -f difference.jq
Here is a solution which uses --argfile and project/1 from pull/1062
def project(q):
. as $in
| reduce (q | if type == "object" then keys[] else .[] end) as $k (
{}
; . + { ($k) : ($in[$k]) }
)
;
map(
reduce $arg[] as $a (
.
; select(project($a) != $a)
)
| values
)
If you place the "second" file in second.json, the data in data.json and the above filter in filter.jq you can run this with
jq -M --argfile arg second.json -f filter.jq data.json
to produce
[
{
"name": "Elise",
"surname": "Long",
"isActive": false,
"balance": "$1,892.72"
},
{
"name": "Hyde",
"surname": "Adkins",
"isActive": true,
"balance": "$1,769.34"
},
{
"name": "Matthews",
"surname": "Jefferson",
"isActive": true,
"balance": "$1,991.42"
}
]
You can replace the expression select(project($a) != $a) with something else if you want to revise the equality criteria for the objects.
Thinking about this a little more we can eliminate the need for project/1 by using contains. This should be more efficient as it eliminates construction of a temporary object.
map(
reduce $arg[] as $a (
.
; select(.!=null and contains($a)==false)
)
| values
)
this can be further simplified using any:
map(select(any(.; contains($arg[]))==false))
which is short enough to be used directly on the command line:
jq -M --argfile arg second.json 'map(select(any(.; contains($arg[]))==false))' data.json
jq solution:
jq --slurpfile s f2.json '[ .[] | . as $o | if (reduce $s[0][] as $i
([]; . + [($o | contains($i))]) | any) then empty else $o end ]' f1.json
The output:
[
{
"name": "Elise",
"surname": "Long",
"isActive": false,
"balance": "$1,892.72"
},
{
"name": "Hyde",
"surname": "Adkins",
"isActive": true,
"balance": "$1,769.34"
},
{
"name": "Matthews",
"surname": "Jefferson",
"isActive": true,
"balance": "$1,991.42"
}
]
this is my data structure:
[
{
"name": "name1",
"organizations": [
{
"name": "name2",
"spaces": [
{
"name": "name3",
"otherkey":"otherval"
},
{
"name": "name4",
"otherkey":"otherval"
}
]
}
]
},
{
"name": "name21",
"organizations": [
{
"name": "name22",
"spaces": [
{
"name": "name23",
"otherkey":"otherval"
},
{
"name": "name24",
"otherkey":"otherval"
}
]
}
]
}
]
i just want to keep name=name1, remove the nested array object with name=name4 and want to keep the rest of the object intact. I tried with map(select) but this will just give me the full object. Is it possible to work with del on specific subarrays and keep the rest as it is?
result should be the following. in addition i want to avoid enumeration all attributes to keep on outer objects:
[
{
"name": "name1",
"organizations": [
{
"name": "name2",
"spaces": [
{
"name": "name3",
"otherkey":"otherval"
}
]
}
]
}
]
any idea? thanks!
A very targeted solution would be:
path(.[0].organizations[0].spaces) as $target
| (getpath($target) | map(select(.name != "name4"))) as $new
| setpath($target; $new)
If permissible, though, you might consider:
walk(if type == "object" and .spaces|type == "array"
then .spaces |= map(select(.name != "name4"))
else . end)
or:
del(.. | .spaces? // empty | .[] | select(.name == "name4") )
(If your jq does not have walk/1 then its jq definition can easily be found by googling.)
You can use the below and it will remove the "name": "name4" array only.
jq 'del(.[] | .organizations? | .[] | .spaces?|.[] | select(.name? == "name4"))' yourJsonFile.json
Here is a solution using select, reduce, tostream and delpaths
map(
select(.name == "name1")
| reduce (tostream|select(length==2)) as [$p,$v] (
.
; if [$p[-1],$v] == ["name","name4"] then delpaths([$p[:-1]]) else . end
)
)
I took a similar approach as #peak but inverted it, so instead of selecting what you want and setting that in the output we're selecting what we don't want and deleting it.
[path(.organizations[0].spaces[]| select(.name == "name4")] as $trash | delpaths($trash)
So my objective is to merge json files obtain this format:
{
"title": "NamesBook",
"list": [
{
"name": "Ajay"
},
{
"name": "Al"
}
]
}
And I have files that look like this format:
blahblah.json
{
"title": "NamesBook",
"list": [
{
"name": "Ajay"
}
]
}
blueblue.json
{
"title": "NamesBook",
"list": [
{
"name": "Al"
}
]
}
I can store the list array of all my names in a variable with the following:
x = jq -s '.[].list' *.json
And then I was planning on appending the variable to an empty array in a file I created, out.json, which looks like this:
{
"type": "NamesBook",
"list": []
}
However, when my script runs over the line
jq '.list[] += "$x"' out.json'
It brings up a jq error:
Cannot iterate over null.
Even when I add a random element, the same error shows up. Tips on how I should proceed? Are there other tools in jq to help achieve merging arrays?
Let me also provide just what the title asks for, because I'm sure a lot of people that stepped on this question look for something simpler.
Any of the following (added math2001 and pmf answers):
echo -e '["a","b"]\n["c","d"]' | jq -s 'add'
echo -e '["a","b"]\n["c","d"]' | jq -s 'flatten(1)'
echo -e '["a","b"]\n["c","d"]' | jq -s 'map(.[])'
echo -e '["a","b"]\n["c","d"]' | jq -s '[.[][]]'
echo -e '["a","b"]\n["c","d"]' | jq '.[]' | jq -s
results in:
[
"a",
"b",
"c",
"d"
]
Note: Also any of the above can apply to arrays of objects.
You can merge your files with add (jq 1.3+):
jq -s '.[0].list=[.[].list|add]|.[0]' *.json
or flatten (jq 1.5+):
jq -s '.[0].list=([.[].list]|flatten)|.[0]' *.json
[.[].list] - creates an array of all "list" arrays
[
[
{
"name": "Ajay"
}
],
[
{
"name": "Al"
}
]
]
[.[].list]|flatten - flatten it (or .[].list|add - add all the arrays together)
[
{
"name": "Ajay"
},
{
"name": "Al"
}
]
.[0].list=([.[].list]|flatten)|.[0] - replace the first "list" with the merged one, output it.
{
"title": "NamesBook",
"list": [
{
"name": "Ajay"
},
{
"name": "Al"
}
]
}
Assuming every file will have the same title and you're simply combining the list contents, you could do this:
$ jq 'reduce inputs as $i (.; .list += $i.list)' blahblah.json blueblue.json
This just takes the first item and adds to its list, the list of all the other inputs.
The OP did not specify what should happen if there are objects for which .title is not "NamesBook". If the intent is to select objects with .title equal to "NamesBook", one could write:
map(select(.title == "NamesBook"))
| {title: .[0].title, list: map( .list ) | add}
This assumes that jq is invoked with the -s option.
Incidentally, add is the way to go here: simple and fast.
I want to use jq to remove all objects with a given name from all arrays in the input data. For example deleting "Name1" from this:
{
"Category1": [
{
"name": "Name1",
"desc": "Desc1"
},
{
"name": "Name2",
"desc": "Desc2"
}
],
"Category2": [
{
"name": "Name1",
"desc": "Desc1"
},
{
"name": "Name3",
"desc": "Desc3"
}
],
"Category3": [
{
"name": "Name4",
"desc": "Desc4"
}
]
}
Should yield this:
{
"Category1": [
{
"name": "Name2",
"desc": "Desc2"
}
],
"Category2": [
{
"name": "Name3",
"desc": "Desc3"
}
],
"Category3": [
{
"name": "Name4",
"desc": "Desc4"
}
]
}
I haven't worked with jq, or indeed JSON, much and after several hours of googling and experimenting I haven't been able to figure it out. How would I do this?
The closest I managed was this:
cat input | jq 'keys[] as $k | .[$k] |= map( select( .name != "Name1"))'
This does filter each of the arrays but returns the result as three separate objects and this is not what I want.
If the structure of your input JSON is always as seen on your example, try this:
map_values(map(select(.name != "Name1")))
Here is a solution that will remove all objects with the specified name, wherever they occur. It uses the generic function walk/1,
which is a built-in in versions of jq > 1.5, and can therefore be omitted if your jq includes it, but there is no harm in including it redundantly, e.g. in a jq script.
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
walk(if type == "object" and .name == "Name1" then empty else . end)
If you really only want to remove objects from arrays, then you could use:
walk(if type == "array" then map(select( type != "object" or .name != "Name1")) else . end)
Here is a solution which uses reduce and del
reduce keys[] as $k (
.
; del(.[$k][] | select(.name == "Name1"))
)