Parse netdata json output (mulltiple arrays) with jq - arrays

I'm trying to use jq to combine two arrays and running into a bit of trouble.
I'm attempting to parse out the data from netdata (netdata.firehol.org) and the two pieces of data within the json response that I"m interested in are both part of an array. The first array is labels for the datapoints in the second array.
Sample Input
[
"time",
"guest_nice",
"guest",
"steal",
"softirq",
"irq",
"user",
"system",
"nice",
"iowait"
]
[
1460728600,
0,
0,
0,
0.45731,
0,
0.25108,
11.74702,
48.22465,
0
]
Input
If you want to grab fresh data yourself to test against, you can use the following:
curl -s -X GET --header 'Accept: application/json'
'http://netdata.firehol.org/api/v1/data?chart=system.cpu&after=-10&before=0&points=1&group=average&format=json&options=seconds%2Cjsonwrap' | jq '.result.labels, .result.data[]'
I've tried to use map() as well as trying to assign vars to both arrays and then print out the objects together, but have been unsuccessful (below).
Code
| jq '.result.labels as $labels | .result.data[] as $data | .result.data[] | Label: $labels[.], data: $data[.]}'
I appreciate anyone's insight in advance as I'm a little stuck, and would prefer to be able to do this all in jq rather than using for loops in bash (if possible).
Expected Ouput
{
"time": "1460728600",
"guest_nice": "0",
...
}

You haven't specified exactly how you want the arrays to be combined, but one approach is to use transpose, which in this case is effectively a kind of zip. For example:
$ jq -n -c '[["a","b"], [1,2]] | transpose'
yields: [["a",1],["b",2]]
If you wanted an array of objects, then with the same input,
transpose | map( { (.[0]) : .[1] } )
would yield: [{"a":1},{"b":2}]
If your jq does not have transpose, here is its definition:
# transpose a possibly jagged matrix, quickly;
# rows are padded with nulls so the result is always rectangular.
def transpose:
[range(0; (map(length) | max)) as $j
| [range(0; length) as $i | .[$i][$j] ] ] ;
Alternatively, if you would prefer a very brief zip:
def zip: [range(0; .[0]|length) as $i | [.[0][$i], .[1][$i]]];

Here is a solution that handles the general case where the first array contains the key names and the following arrays contain values using transpose and from_entries
{h:.[0], v:.[1:][]} # {h:[keys], v:[values]}
| [.h, .v] # [ [keys], [values] ] ...
| [ transpose[] | {key:.[0], value:.[1]} ] # [ {"key":key, "value":value}, ... ]
| from_entries # { key:value, key:value, ... }
For example, if this filter is in filter.jq and data.json contains
["time","guest_nice","guest","steal","softirq","irq","user","system","nice","iowait"]
[1460728600,0,0,0,0.45731,0,0.25108,11.74702,48.22465,0]
[1460728601,0,0,0,0.45732,0,0.25109,12.74703,49,0]
then the command
jq -M -s -c -f filter.jq data.json
produces
{"time":1460728600,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45731,"irq":0,"user":0.25108,"system":11.74702,"nice":48.22465,"iowait":0}
{"time":1460728601,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45732,"irq":0,"user":0.25109,"system":12.74703,"nice":49,"iowait":0}

Related

How to limit properties of a JSON object given array of property names using JQ?

Assuming I have the following JSON object (which is just an example):
{
"foo": 1,
"bar": 2,
"baz": 3
}
And the following JSON array (another example):
["foo", "baz"]
How could I use jq to output the following object?
{
"foo": 1,
"baz": 3
}
I hate asking this question because I feel certain it has been answered before, but google has failed me and my jq-fu is just not where it needs to be to figure this out.
Using a reduce to iteratively build up the result object would be one way:
echo '["foo", "baz"]' | jq --argjson index '{"foo":1,"bar":2,"baz":3}' '
reduce .[] as $x ({}; .[$x] = $index[$x])
'
Using JOIN, creating key-value pairs, and employing from_entries for assembly would be another way:
echo '["baz", "foo"]' | jq --argjson index '{"foo":1,"bar":2,"baz":3}' '
JOIN($index; .) | map({key:.[0], value:.[1]}) | from_entries
'
Output:
{
"foo": 1,
"baz": 3
}
Provided that . is the object and $arr is the array, the following does the trick
delpaths(keys - $arr | map([.]))
To achieve the desired result, one could write:
jq '{foo, baz}'
This can (with some considerable trepidation) be made into a solution for the given problem by text wrangling, e.g. along the lines of:
jq "$(echo '["foo", "baz"]' | jq -Rr '"{" + .[1:-1] + "}" ')"
or
jq "$(echo '["foo", "baz"]' | sed -e 's/\[/{/' -e 's/\]/}/')"
Here's a reduce-free solution that assumes $keys is the array of keys of interest and that is possibly more efficient than the one involving array subtraction:
. as $in | INDEX( $keys[]; $in[.] )

Split string array evenly into sub arrays whilst using a filter with jq

Given I have the following json
[
"/home/test-spa/src/components/modals/super-admin/tests/integration/index.test.tsx",
"/home/test-spa/src/components/modals/delete-user/tests/index.test.tsx",
"/home/test-spa/src/components/modals/edit-admin/tests/integration/index.test.tsx",
"/home/test-spa/src/components/modals/delete-admin/tests/index.test.tsx",
"/home/test-spa/src/components/modals/add-user/tests/integration/index.test.tsx",
"/home/test-spa/src/components/modals/add-admin/tests/integration/index.test.tsx",
"/home/test-spa/src/components/modals/edit-user/tests/index.test.tsx",
"/home/test-spa/src/components/modals/change-user/tests/index.test.tsx",
"/home/test-spa/src/other-directory/modals/tests/index.test.ts",
"/home/test-spa/src/directory/modals/tests/index.test.ts",
]
I want to exclude anything that has directory or other-directory in the string
I then want to split the array into 4 arrays but I want to evenly split anything that has integration in the string i.e I don't want all the integrations in one array. Any other string can then be split across the 4 arrays.
I want to use jq in order to perform this filter. The following code allows me to split the json into 4 but does not do the required filtering as outlined above.
jq -cM '[_nwise(length / 4 | floor)]'
As a result I am looking for something like the following output (as long as the integration tests are split as evenly as possible, the other strings can then fill in evenly and order does not matter)
[
[
"/home/test-spa/src/components/modals/super-admin/tests/integration/index.test.tsx",
"/home/test-spa/src/components/modals/delete-user/tests/index.test.tsx"
],
[
"/home/test-spa/src/components/modals/edit-admin/tests/integration/index.test.tsx",
"/home/test-spa/src/components/modals/delete-admin/tests/index.test.tsx"
],
[
"/home/test-spa/src/components/modals/add-admin/tests/integration/index.test.tsx",
"/home/test-spa/src/components/modals/edit-user/tests/index.test.tsx"
],
[
"/home/test-spa/src/components/modals/add-user/tests/integration/index.test.tsx",
"/home/test-spa/src/components/modals/change-user/tests/index.test.tsx"
]
]
If the number of buckets is predetermined
Here's a generic "round-robin" function, written so that the distribution of the "has" and "has not" strings can be performed efficiently (i.e., without concatenating any arrays):
# s is a stream, $n a predetermined number of buckets
def roundrobin(s; $n):
reduce s as $s ({n: 0, a: []}; .a[.n % $n] += [$s] | .n+=1) | .a;
# First exclude the unwanted elements:
map(select(test("(other-)?directory")|not))
# Perform the required round-robin:
| roundrobin( (.[] | select(index("integration"))),
(.[] | select(index("integration")|not)); 4)
If the number of buckets is data-driven
If the number of buckets should depend on the number of occurrences of the specified string, then using the above-defined roundrobin filter, a reasonably efficient solution could be written as follows:
# First exclude the unwanted elements:
map(select(test("(other-)?directory")|not))
# Form an array of the strings with the specified substring
| map(select(index("integration"))) as $has
# Perform the required round-robin:
| roundrobin( $has[], ((.-$has)[]); $has|length)
Here's what I came up with, to split into N buckets:
def bucket_shift($n):
# loop through all input, shift each elem into bucket
reduce .[] as $elem ( { count: 0, rv: [] };
(.rv[(.count % $n)] += [$elem] | .count += 1))
| .rv ;
# get rid of everything with directory or other-directory
[ .[] | select(test("directory|other-directory") | not) ]
# grab all lines with "integration" in an array
| [ ([ .[] | select(test("integration")) ]),
# grab all lines without "integration" into a second array
([ .[] | select(test("integration") | not) ]) ]
# flatten and divide into buckets (arg passed in)
| flatten | bucket_shift($num_buckets|tonumber)
I labeled each line in your input so that I could track them more easily, then added a couple of extra lines so that the results were not evenly divisible by the number of buckets you wanted, to make sure it would balance well. Lines I and J should be filtered out.
<~> $ jq . /tmp/so.json
[
"A/home/test-spa/src/components/modals/super-admin/tests/integration/index.test.tsx",
"B/home/test-spa/src/components/modals/delete-user/tests/index.test.tsx",
"C/home/test-spa/src/components/modals/edit-admin/tests/integration/index.test.tsx",
"D/home/test-spa/src/components/modals/delete-admin/tests/index.test.tsx",
"E/home/test-spa/src/components/modals/add-user/tests/integration/index.test.tsx",
"F/home/test-spa/src/components/modals/add-admin/tests/integration/index.test.tsx",
"G/home/test-spa/src/components/modals/edit-user/tests/index.test.tsx",
"H/home/test-spa/src/components/modals/change-user/tests/index.test.tsx",
"IX/home/test-spa/src/other-directory/modals/tests/index.test.ts",
"JX/home/test-spa/src/directory/modals/tests/index.test.ts",
"K/home/test-spa/src/components/modals/change-user/tests/index.test.tsx",
"L/home/test-spa/src/components/modals/add-admin/tests/integration/index.test.tsx"
]
The script as above:
<~> $ cat /tmp/so.jq
def bucket_shift($n):
# loop through all input, shift each elem into bucket
reduce .[] as $elem ( { count: 0, rv: [] };
(.rv[(.count % $n)] += [$elem] | .count += 1))
| .rv ;
# get rid of everything with directory or other-directory
[ .[] | select(test("directory|other-directory") | not) ]
# grab all lines with "integration" in an array
| [ ([ .[] | select(test("integration")) ]),
# grab all lines without "integration" into a second array
([ .[] | select(test("integration") | not) ]) ]
# flatten and divide into buckets (arg passed in)
| flatten | bucket_shift($num_buckets|tonumber)
Break out into 4 buckets:
<~> $ jq --arg num_buckets 4 -f /tmp/so.jq /tmp/so.json
[
[
"A/home/test-spa/src/components/modals/super-admin/tests/integration/index.test.tsx",
"L/home/test-spa/src/components/modals/add-admin/tests/integration/index.test.tsx",
"H/home/test-spa/src/components/modals/change-user/tests/index.test.tsx"
],
[
"C/home/test-spa/src/components/modals/edit-admin/tests/integration/index.test.tsx",
"B/home/test-spa/src/components/modals/delete-user/tests/index.test.tsx",
"K/home/test-spa/src/components/modals/change-user/tests/index.test.tsx"
],
[
"E/home/test-spa/src/components/modals/add-user/tests/integration/index.test.tsx",
"D/home/test-spa/src/components/modals/delete-admin/tests/index.test.tsx"
],
[
"F/home/test-spa/src/components/modals/add-admin/tests/integration/index.test.tsx",
"G/home/test-spa/src/components/modals/edit-user/tests/index.test.tsx"
]
]
Break into 3 buckets, instead:
<~> $ jq --arg num_buckets 3 -f /tmp/so.jq /tmp/so.json
[
[
"A/home/test-spa/src/components/modals/super-admin/tests/integration/index.test.tsx",
"F/home/test-spa/src/components/modals/add-admin/tests/integration/index.test.tsx",
"D/home/test-spa/src/components/modals/delete-admin/tests/index.test.tsx",
"K/home/test-spa/src/components/modals/change-user/tests/index.test.tsx"
],
[
"C/home/test-spa/src/components/modals/edit-admin/tests/integration/index.test.tsx",
"L/home/test-spa/src/components/modals/add-admin/tests/integration/index.test.tsx",
"G/home/test-spa/src/components/modals/edit-user/tests/index.test.tsx"
],
[
"E/home/test-spa/src/components/modals/add-user/tests/integration/index.test.tsx",
"B/home/test-spa/src/components/modals/delete-user/tests/index.test.tsx",
"H/home/test-spa/src/components/modals/change-user/tests/index.test.tsx"
]
]
To have a default bucket size, you can do something like this:
bucket_shift($ARGS.named["num_buckets"] // 4|tonumber)

Convert bash array to json array and insert to file using jq

Given a bash array, how to convert it to a JSON array in order to output to a file with jq?
Additionnally: is there a way to keep the server_nohup array unchanged instead of re-writing the whole json file each time?
newArray=(100 200 300)
jq -n --arg newArray $newArray '{
client_nohup: [
$newArray
],
server_nohup: [
]
}' > $projectDir/.watch.json
Current output:
{
"client_nohup": [
"100"
],
"server_nohup": []
}
Desired output:
{
"client_nohup": [
100,
200,
300
],
"server_nohup": []
}
(1) If all the values in newArray are valid as JSON values without spaces, then you could get away with piping the values as a stream, e.g.
newArray=(100 200 300)
echo "${newArray[#]}" |
jq -s '{client_nohup: ., server_nohup: []}'
(2)
Now let's suppose you merely wish to update the "nohup" object in a file, say nohup.json:
{ "client_nohup": [], "server_nohup": [ "keep me" ] }
Since you are using bash, you can then write:
echo "${newArray[#]}" |
jq -s --argjson nohup "$(cat nohup.json)" '
. as $newArray | $nohup | .client_nohup = $newArray
'
Output
(1)
{
"client_nohup": [
100,
200,
300
],
"server_nohup": []
}
(2)
{
"client_nohup": [
100,
200,
300
],
"server_nohup": [
"keep me"
]
}
Other cases
Where there's a will, there's a jq way :-)
See for example the accepted answer at How to format a bash array as a JSON array (though this is not a completely generic solution).
For a generic solution, see 𝑸: How can a variable number of arguments be passed to jq? How can a bash array of values be passed in to jq as a single argument? at the jq FAQ https://github.com/stedolan/jq/wiki/FAQ
Generic Solutions
To be clear, if the array values are known to be valid JSON, there are several good options; if the array values are arbitrary bash strings, then the only efficient, generic way to handle them with jq is by using the -R jq option (e.g. in conjunction with -s), but then the (bash) strings will all be read in as JSON strings, so any intended type information will be lost. (The point here hinges on the technicality that bash strings cannot CONTAIN NUL characters.)
Often, to alleviate the latter concern, one can convert numeric strings to JSON numbers, e.g. using the jq idiom: (tonumber? // .).
In general, the only truly safe way to do this is with multiple invocations of jq, adding each element to the output of the previous command.
arr='[]' # Empty JSON array
for x in "${newArray[#]}"; do
arr=$(jq -n --arg x "$x" --argjson arr "$arr" '$arr + [$x]')
done
This ensures that each element x of your bash array is properly encoded prior to adding it to the JSON array.
This is complicated, though, by the fact that bash doesn't not distinguish between numbers and strings. This encodes your array as ["100", "200", "300"], not [100, 200, 300]. In the end, you need to have some awareness of what your array contains, and preprocess it accordingly.

jq won't allow to iterate over results

In code below I'm trying to match values from a bash array with values from json array using jq.
PROJECTS=$(curl -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "${GITLAB_URL}/api/v4/projects")
for GITLAB_TAG in "${GITLAB_TAGS[#]}"; do
PROJECTS=`echo "${PROJECTS}" \
| jq --arg gitlab_tag "$GITLAB_TAG" '[ .[] | select(.tag_list[] | contains($gitlab_tag)) ]'`
done
PROJECTS=$(echo "$PROJECTS" | jq -r '.[]')
Consider the following JSON payload as a sample API response:
[{"id":31,"description":"","default_branch":null,"tag_list":["dev","app"],"archived":false,"visibility":"private"},{"id":28,"description":"","default_branch":"master","tag_list":["dev","app"],"archived":false,"visibility":"private"}]
This works only partially, because I can't iterate over results ($PROJECTS).
printf "${PROJECTS[0]}" prints the whole array.
Am I missing something here?
It appears that you should be using index/1 to check .tag_list (not .tag_list[]):
[ .[] | select(.tag_list | index($gitlab_tag))
jq has no knowledge of bash array variables. For these, you therefore have two basic options: 1) use bash to iterate over the bash array variable (calling jq as often as necessary); 2) present the contents of the bash array variable in a way that jq can handle.
There are many variations of (2). Consider for example:
$ a=(a "b c")
$ printf "%s\n" "${a[#]}" | jq -R | jq -s
[
"a",
"b c"
]

Newbie looking to learn nested JSON

I just started learning JSON and how to convert back and forth between different file types.
If I'm trying to convert a CSV file to JSON
Say if I have a CSV file of people
(First name, Last name, age)
David Chin 40
David Lin 40
David Ping 30
Sandra Lee 25
Sandra Long 45
I can convert that to JSON without any formatting.
How would I go about trying to convert the CSV to a nested JSON?
Hopefully output would look like
{"David":{
"Chin": "40"
"Lin": "40"
"Ping": "30"
},
"Sandra":{
"Lee": "25"
"Long": "45"}
}
Is there any particular code where I can freely format the conversion?
Any help toward the right direction is appreciated or pointers on how to format JSON would also work. I just don't know where I would manipulate to get that format in the output.
Here is a solution using jq
If the file filter.jq contains
[
split("\n") # split string into lines
| [.[0] | split(",")[]] as $headers # split header
| (.[1:][] | split(",")) # split data
| select(length>0) # eliminate blanks
| [ [ $headers, . ] #
| transpose[] # assemble objects
| {key:.[0], value:.[1]} # from keys and values
] | from_entries #
]
| reduce group_by(.First)[][] as $r ( # construct final result
{}
; .[$r.First] = (.[$r.First]//{}) * {($r.Last):$r.Age}
)
and data contains
First,Last,Age
David,Chin,40
David,Lin,40
David,Ping,30
Sandra,Lee,25
Sandra,Long,45
then the command
jq -M -R -s -r -f filter.jq data
will produce the output
{
"David": {
"Chin": "40",
"Lin": "40",
"Ping": "30"
},
"Sandra": {
"Lee": "25",
"Long": "45"
}
}

Resources