Let's say I have more namespaces with the similar k8s resource (some might have different images used). I am trying to get .metadata.namespace using jq from the following json object (let's call it test.json):
{
"items": [
{
"metadata": {
"name": "app",
"namespace": "test1"
},
"spec": {
"components": [
{
"database": {
"from": "service",
"value": "redis"
},
"image": "test.com/lockmanager:1.1.1",
"name": "lockmanager01",
"replicas": 2,
"type": "lockmanager"
},
{
"database": {
"from": "service",
"value": "postgresql"
},
"image": "test.com/jobmanager:1.1.1",
"name": "jobmanager01",
"replicas": 2,
"type": "jobmanager"
}
]
}
}
]
}
if following condition is met:
.spec.components[].type == "jobmanager" and .spec.components[].image != "test.com/jobmanager:1.1.1"
but can't find the correct statement.
I tried:
cat test.json | jq '.items[] | select((.spec.components[].name? | contains("jobmanager01")) and (.spec.components[].image != "test.com/jobmanager:1.1.1")) | .metadata.namespace''
but it returns all namespaces and, moreover, those I am interested in (because I know they contain different image), are returned twice.
Please advise what am I doing wrong?
You state that the selection criterion is:
.spec.components[].type == "jobmanager" and
.spec.components[].image != "test.com/jobmanager:1.1.1"
but that does not make much sense, given the semantics of .[].
I suspect you meant that you want to select items from .spec.components such that
.type == "jobmanager" and .image != "test.com/jobmanager:1.1.1"
If that's the case, you could use any, so that your query would look like this:
.items[]
| select( any(.spec.components[];
(.name? | contains("jobmanager01")) and
.image != "test.com/jobmanager:1.1.1") )
| .metadata.namespace
all distinct
If you want all the distinct .namespace values satisfying the condition, you could go with:
[.items[]
| .metadata.namespace as $it
| .spec.components[]
| select( (.name? | contains("jobmanager01")) and
.image != "test.com/jobmanager:1.1.1" )
| $it]
| unique[]
Efficient version of "all-distinct" solution
To avoid unnecessary checks, if .namespace is always a string, we could write:
reduce .items[] as $item ({};
$item.metadata.namespace as $it
| if .[$it] then . # already seen
elif any( $item.spec.components[];
((.name? | contains("jobmanager01")) and
.image != "test.com/jobmanager:1.1.1") )
then .[$it] = true
else . end )
| keys_unsorted[]
Related
I have three operations with jq to get the right result. How can I do it within one command?
Here is a fragment from the source JSON file
[
{
"Header": {
"Tenant": "tenant-1",
"Rcode": 200
},
"Body": {
"values": [
{
"id": "0b0b-0c0c",
"name": "NumberOfSearchResults"
},
{
"id": "aaaa0001-0a0a",
"name": "LoadTest"
}
]
}
},
{
"Header": {
"Tenant": "tenant-2",
"Rcode": 200
},
"Body": {
"values": []
}
},
{
"Header": {
"Tenant": "tenant-3",
"Rcode": 200
},
"Body": {
"values": [
{
"id": "cccca0003-0b0b",
"name": "LoadTest"
}
]
}
},
{
"Header": {
"Tenant": "tenant-4",
"Rcode": 200
},
"Body": {
"values": [
{
"id": "0f0g-0e0a",
"name": "NumberOfSearchResults"
}
]
}
}
]
I apply two filters and create two intermediate JSON files. First I create the list of all tenants
jq -r '[.[].Header.Tenant]' source.json >all-tenants.json
And then I select to create an array of all tenants not having a particular key present in the Body.values[] array:
jq -r '[.[] | select (all(.Body.values[]; .name !="LoadTest") ) ] | [.[].Header.Tenant]' source.json >filter1.json
Results - all-tenants.json
["tenant-1",
"tenant-2",
"tenant-3",
"tenant-4"
]
filter1.json
["tenant-2",
"tenant-4"
]
And then I substruct filter1.json from all-tenants.json to get the difference:
jq -r -n --argfile filter filter1.json --argfile alltenants all-tenants.json '$alltenants - $filter|.[]'
Result:
tenant-1
tenant-3
Tenant names - values for the "Tenant" key are unique and each of them occurs only once in the source.json file.
Just to clarify - I understand that I can have a select condition(s) that would give me the same resut as subtracting two arrays.
What I want to understand - how can I assign and use these two arrays into vars directly in a single command not involving the intermediate files?
Thanks
Use your filters to fill in the values of a new object and use the keys to refer to the arrays.
jq -r '{
"all-tenants": [.[].Header.Tenant],
"filter1": [.[]|select (all(.Body.values[]; .name !="LoadTest"))]|[.[].Header.Tenant]
} | .["all-tenants"] - .filter1 | .[]'
Note: .["all-tenants"] is required by the special character "-" in that key. See the entry under Object Identifier-Index in the manual.
how can I assign and use these two arrays into vars directly in a single command not involving the intermediate files?
Simply store the intermediate arrays as jq "$-variables":
[.[].Header.Tenant] as $x
| ([.[] | select (all(.Body.values[]; .name !="LoadTest") ) ] | [.[].Header.Tenant]) as $y
| $x - $y
If you want to itemize the contents of $x - $y, then simply add a final .[] to the pipeline.
input
{
"apps": [
{
"name": "whatever1",
"id": "ID1"
},
{
"name": "whatever2",
"id": "ID2",
"dep": [
"a.jar"
]
},
{
"name": "whatever3",
"id": "ID3",
"dep": [
"a.jar",
"b.jar"
]
}
]
}
output
{
"apps": [
{
"name": "whatever1",
"id": "ID1",
"dep": [
"b.jar"
]
},
{
"name": "whatever2",
"id": "ID2",
"dep": [
"a.jar",
"b.jar"
]
},
{
"name": "whatever3",
"id": "ID3",
"dep": [
"a.jar",
"b.jar"
]
}
]
}
in the above example
whatever1 does not have dep, so create one.
whatever2 has dep and does not have b.jar, so add b.jar
whatever3 aready has dep and b.jar is there so untouched.
what i have tried.
# add blindly, whatever3 is not right
cat dep.json | jq '.apps[].dep += ["b.jar"]'
# missed one level and whatever3 is gone.
cat dep.json | jq '.apps | map(select(.dep == null or (.dep | contains(["b.jar"]) | not)))[] | .dep += ["b.jar"]'
For the sake of clarity, let's define a helper function for performing the core task:
# It is assumed that the input is an object
# that either does not have the specified key or
# that it is array-valued
def ensure_has($key; $value):
if has($key) and (.[$key] | index($value)) then .
else .[$key] += [$value]
end ;
The task can now be accomplished in a straightforward way:
.apps |= map(ensure_has("dep"; "b.jar"))
Alternatively ...
.apps[] |= ensure_has("dep"; "b.jar")
after some trial and error, it looks like this is one way to do it.
cat dep.json | jq '.apps[].dep |= (. + ["b.jar"] | unique)'
Is there a way to compare two json files in jq? Specifically, I'd like to be able to remove objects from one json file if they occur in another json file. Basically, subtract one file from another. It would be a bonus if I could generalize this so that I could define the equality criteria for the objects, but this is not strictly necessary, it can be based strictly on the objects being identical.
So the more general case would look like this. Let's say I have a file that looks like this:
[
{
"name": "Cynthia",
"surname": "Craig",
"isActive": true,
"balance": "$2,426.88"
},
{
"name": "Elise",
"surname": "Long",
"isActive": false,
"balance": "$1,892.72"
},
{
"name": "Hyde",
"surname": "Adkins",
"isActive": true,
"balance": "$1,769.34"
},
{
"name": "Matthews",
"surname": "Jefferson",
"isActive": true,
"balance": "$1,991.42"
},
{
"name": "Kris",
"surname": "Norris",
"isActive": false,
"balance": "$2,137.11"
}
]
And I have a second file that looks like this:
[
{
"name": "Cynthia",
"surname": "Craig"
},
{
"name": "Kris",
"surname": "Norris"
}
]
I'd like to remove any objects from the first file where the name and surname fields match an object of the second file, so that the results should look like this:
[
{
"name": "Elise",
"surname": "Long",
"isActive": false,
"balance": "$1,892.72"
},
{
"name": "Hyde",
"surname": "Adkins",
"isActive": true,
"balance": "$1,769.34"
},
{
"name": "Matthews",
"surname": "Jefferson",
"isActive": true,
"balance": "$1,991.42"
}
]
The following solution is intended to be generic, efficient and as simple as possible subject to the first two objectives.
Genericity
For genericity, let us suppose that $one and $two are two arrays of
JSON entities, and that we wish to find those items, $x, in $one
such that ($x|filter) does not appear in map($two | filter), where filter is an arbitrary filter. (In the present instance, it is {surname, name}.)
The solution uses INDEX/1, which was added to jq after the official 1.5 release, so we begin by reproducing its definition:
def INDEX(stream; idx_expr):
reduce stream as $row ({};
.[$row|idx_expr|
if type != "string" then tojson
else .
end] |= $row);
def INDEX(idx_expr): INDEX(.[]; idx_expr);
Efficiency
For efficiency, we will need to use a JSON object as a dictionary;
since keys must be strings, we will need to ensure that when converting an object
to a string, the objects are normalized. For this, we define normalize as follows:
# Normalize the input with respect to the order of keys in objects
def normalize:
. as $in
| if type == "object" then reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | normalize) } )
elif type == "array" then map( normalize )
else .
end;
To construct the dictionary, we simply apply (normalize|tojson):
def todict(filter):
INDEX(filter| normalize | tojson);
The solution
The solution is now quite simple:
# select those items from the input stream for which
# (normalize|tojson) is NOT in dict:
def MINUS(filter; $dict):
select( $dict[filter | normalize | tojson] | not);
def difference($one; $two; filter):
($two | todict(filter)) as $dict
| $one[] | MINUS( filter; $dict );
difference( $one; $two; {surname, name} )
Invocation
$ jq -n --argfile one one.json --argfile two two.json -f difference.jq
Here is a solution which uses --argfile and project/1 from pull/1062
def project(q):
. as $in
| reduce (q | if type == "object" then keys[] else .[] end) as $k (
{}
; . + { ($k) : ($in[$k]) }
)
;
map(
reduce $arg[] as $a (
.
; select(project($a) != $a)
)
| values
)
If you place the "second" file in second.json, the data in data.json and the above filter in filter.jq you can run this with
jq -M --argfile arg second.json -f filter.jq data.json
to produce
[
{
"name": "Elise",
"surname": "Long",
"isActive": false,
"balance": "$1,892.72"
},
{
"name": "Hyde",
"surname": "Adkins",
"isActive": true,
"balance": "$1,769.34"
},
{
"name": "Matthews",
"surname": "Jefferson",
"isActive": true,
"balance": "$1,991.42"
}
]
You can replace the expression select(project($a) != $a) with something else if you want to revise the equality criteria for the objects.
Thinking about this a little more we can eliminate the need for project/1 by using contains. This should be more efficient as it eliminates construction of a temporary object.
map(
reduce $arg[] as $a (
.
; select(.!=null and contains($a)==false)
)
| values
)
this can be further simplified using any:
map(select(any(.; contains($arg[]))==false))
which is short enough to be used directly on the command line:
jq -M --argfile arg second.json 'map(select(any(.; contains($arg[]))==false))' data.json
jq solution:
jq --slurpfile s f2.json '[ .[] | . as $o | if (reduce $s[0][] as $i
([]; . + [($o | contains($i))]) | any) then empty else $o end ]' f1.json
The output:
[
{
"name": "Elise",
"surname": "Long",
"isActive": false,
"balance": "$1,892.72"
},
{
"name": "Hyde",
"surname": "Adkins",
"isActive": true,
"balance": "$1,769.34"
},
{
"name": "Matthews",
"surname": "Jefferson",
"isActive": true,
"balance": "$1,991.42"
}
]
this is my data structure:
[
{
"name": "name1",
"organizations": [
{
"name": "name2",
"spaces": [
{
"name": "name3",
"otherkey":"otherval"
},
{
"name": "name4",
"otherkey":"otherval"
}
]
}
]
},
{
"name": "name21",
"organizations": [
{
"name": "name22",
"spaces": [
{
"name": "name23",
"otherkey":"otherval"
},
{
"name": "name24",
"otherkey":"otherval"
}
]
}
]
}
]
i just want to keep name=name1, remove the nested array object with name=name4 and want to keep the rest of the object intact. I tried with map(select) but this will just give me the full object. Is it possible to work with del on specific subarrays and keep the rest as it is?
result should be the following. in addition i want to avoid enumeration all attributes to keep on outer objects:
[
{
"name": "name1",
"organizations": [
{
"name": "name2",
"spaces": [
{
"name": "name3",
"otherkey":"otherval"
}
]
}
]
}
]
any idea? thanks!
A very targeted solution would be:
path(.[0].organizations[0].spaces) as $target
| (getpath($target) | map(select(.name != "name4"))) as $new
| setpath($target; $new)
If permissible, though, you might consider:
walk(if type == "object" and .spaces|type == "array"
then .spaces |= map(select(.name != "name4"))
else . end)
or:
del(.. | .spaces? // empty | .[] | select(.name == "name4") )
(If your jq does not have walk/1 then its jq definition can easily be found by googling.)
You can use the below and it will remove the "name": "name4" array only.
jq 'del(.[] | .organizations? | .[] | .spaces?|.[] | select(.name? == "name4"))' yourJsonFile.json
Here is a solution using select, reduce, tostream and delpaths
map(
select(.name == "name1")
| reduce (tostream|select(length==2)) as [$p,$v] (
.
; if [$p[-1],$v] == ["name","name4"] then delpaths([$p[:-1]]) else . end
)
)
I took a similar approach as #peak but inverted it, so instead of selecting what you want and setting that in the output we're selecting what we don't want and deleting it.
[path(.organizations[0].spaces[]| select(.name == "name4")] as $trash | delpaths($trash)
I want to use jq to remove all objects with a given name from all arrays in the input data. For example deleting "Name1" from this:
{
"Category1": [
{
"name": "Name1",
"desc": "Desc1"
},
{
"name": "Name2",
"desc": "Desc2"
}
],
"Category2": [
{
"name": "Name1",
"desc": "Desc1"
},
{
"name": "Name3",
"desc": "Desc3"
}
],
"Category3": [
{
"name": "Name4",
"desc": "Desc4"
}
]
}
Should yield this:
{
"Category1": [
{
"name": "Name2",
"desc": "Desc2"
}
],
"Category2": [
{
"name": "Name3",
"desc": "Desc3"
}
],
"Category3": [
{
"name": "Name4",
"desc": "Desc4"
}
]
}
I haven't worked with jq, or indeed JSON, much and after several hours of googling and experimenting I haven't been able to figure it out. How would I do this?
The closest I managed was this:
cat input | jq 'keys[] as $k | .[$k] |= map( select( .name != "Name1"))'
This does filter each of the arrays but returns the result as three separate objects and this is not what I want.
If the structure of your input JSON is always as seen on your example, try this:
map_values(map(select(.name != "Name1")))
Here is a solution that will remove all objects with the specified name, wherever they occur. It uses the generic function walk/1,
which is a built-in in versions of jq > 1.5, and can therefore be omitted if your jq includes it, but there is no harm in including it redundantly, e.g. in a jq script.
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
walk(if type == "object" and .name == "Name1" then empty else . end)
If you really only want to remove objects from arrays, then you could use:
walk(if type == "array" then map(select( type != "object" or .name != "Name1")) else . end)
Here is a solution which uses reduce and del
reduce keys[] as $k (
.
; del(.[$k][] | select(.name == "Name1"))
)