I just started learning JSON and how to convert back and forth between different file types.
If I'm trying to convert a CSV file to JSON
Say if I have a CSV file of people
(First name, Last name, age)
David Chin 40
David Lin 40
David Ping 30
Sandra Lee 25
Sandra Long 45
I can convert that to JSON without any formatting.
How would I go about trying to convert the CSV to a nested JSON?
Hopefully output would look like
{"David":{
"Chin": "40"
"Lin": "40"
"Ping": "30"
},
"Sandra":{
"Lee": "25"
"Long": "45"}
}
Is there any particular code where I can freely format the conversion?
Any help toward the right direction is appreciated or pointers on how to format JSON would also work. I just don't know where I would manipulate to get that format in the output.
Here is a solution using jq
If the file filter.jq contains
[
split("\n") # split string into lines
| [.[0] | split(",")[]] as $headers # split header
| (.[1:][] | split(",")) # split data
| select(length>0) # eliminate blanks
| [ [ $headers, . ] #
| transpose[] # assemble objects
| {key:.[0], value:.[1]} # from keys and values
] | from_entries #
]
| reduce group_by(.First)[][] as $r ( # construct final result
{}
; .[$r.First] = (.[$r.First]//{}) * {($r.Last):$r.Age}
)
and data contains
First,Last,Age
David,Chin,40
David,Lin,40
David,Ping,30
Sandra,Lee,25
Sandra,Long,45
then the command
jq -M -R -s -r -f filter.jq data
will produce the output
{
"David": {
"Chin": "40",
"Lin": "40",
"Ping": "30"
},
"Sandra": {
"Lee": "25",
"Long": "45"
}
}
Related
I am trying to curl some URL which returns a json file, as follows
[
10,
20,
30,
40,
]
now I am trying to save these value in file with a variable assigned to it.
need output in file as
a=10
b=20
c=30
d=40
thanks in advance.
You could use implode to create the letters by array index, which is provided in .key when to_entries is applied to an array.
jq -r 'to_entries[] | "\([.key + 97] | implode)=\(.value)"'
a=10
b=20
c=30
d=40
Demo
To provide individual names, you can introduce an array of names and use transpose to make the alignment:
jq -r '
[["Price", "Amount", "Value", "Tax"], .]
| transpose[] | "\(first)=\(last)"
'
Price=10
Amount=20
Value=30
Tax=40
Demo
Note: The list could also be provided from outside jq:
using a JSON array
jq -r --argjson names '["Price", "Amount", "Value", "Tax"]' \
'[$names, .] | transpose[] | "\(first)=\(last)"'
using positional parameters
jq -r '[$ARGS.positional, .] | transpose[] | "\(first)=\(last)"' \
--args Price Amount Value Tax
I have some JSON data which is pretty typical CSV-style data, however it's represented in JSON. I am struggling to figure out the correct jq expression to convert the following JSON back to some JSON which can generate the appropriate CSV with #csv.
There's a fixed number of 'columns', i.e. the "AAA" values, but the number of values in each 'column' is dynamic yet fixed across columns. That is, the length of the arrays "AAA", "BBB", "CCC", etc are all the same, but the length is dynamic and can change between data sets.
Input (note invalid numbers present, to illustrate example):
{
"AAA": [
111.1,
111.2,
111.3,
111..,
111.n
],
"BBB": [
222.1,
222.2,
222.3,
222..,
222.n
],
"CCC": [
333.1,
333.2,
333.3,
333..,
333.n
],
"DDD": [
444.1,
444.2,
444.3,
444..,
444.n
],
"EEE": [
555.1,
555.2,
555.3,
555..,
555.n
]
}
Desired output (note invalid numbers present, to illustrate example):
{
[
"AAA",
"BBB",
"CCC",
"DDD",
"EEE"
],
[
111.1,
222.1,
333.1,
444.1,
555.1
],
[
111.2,
222.2,
333.2,
444.2,
555.2
],
[
111.3,
222.3,
333.3,
444.3,
555.3
],
[
111..,
222..,
333..,
444..,
555..
],
[
111.n,
222.n,
333.n,
444.n,
555.n
]
}
Here is the desired CSV, for illustration purposes (as converting with #csv is pretty straightforward):
AAA,BBB,CCC,DDD,EEE
111.1,222.1,333.1,444.1,555.1
111.2,222.2,333.2,444.2,555.2
111.3,222.3,333.3,444.3,555.3
111..,222..,333..,444..,555..
111.n,222.n,333.n,444.n,555.n
If the required expression is far easier without the first array in the result object containing the "AAA" 'header' values then I can easily live without them.
Thank you.
You can use the transpose function in jq to do the transposing of arrays, formed from keys/values.
jq '[ to_entries[] | [.key, .value[]] ] | transpose'
The bulk of the magic is performed by the transpose built-in, but before that you just need to collect the values into an array of arrays. The CSV result can be generated with the #csv function.
jq --raw-output '[ to_entries[] | [.key, .value[]] ] | transpose[] | #csv'
You could also use map() and be avoid the redundant [..]
jq 'to_entries | map([.key, .value[]]) | transpose'
jq --raw-output 'to_entries | map([.key, .value[]]) | transpose[] | #csv'
I have a dictionary that looks like this:
{
"uid": "d6fc3e2b-0001a",
"name": "ABC Mgmt",
"type": "host"
}
{
"uid": "d6fc3e2b-0002a",
"name": "Server XYZ",
"type": "group"
}
{
"uid": "d6fc3e2b-0003a",
"name": "NTP Primary",
"type": "host"
}
{
"uid": "d6fc3e2b-0004a",
"name": "H-10.10.10.10",
"type": "host"
}
Then I have a txt file:
"d6fc3e2b-0001a"
"d6fc3e2b-0001a","d6fc3e2b-0002a","d6fc3e2b-0003a"
"d6fc3e2b-0004a"
Expected Output:
"ABC Mgmt"
"ABC Mgmt","Server XYZ","NTP Primary"
"H-10.10.10.10"
I have some trouble to make jq using an array which is not json format. I tried various solutions that I found, but none of them worked. I am rather new to scripting, need some help.
input=file.txt
while IFS= read -r line
do
{
value=$(jq -r --arg line "$line" \
'from_entries | .[($line | split(","))[]]' \
dictionary.json)
echo $name
}
done < "$input"
In the following solution, the dictionary file is read using the --slurpfile command-line option, and the lines of "text" are read using inputs in conjunction with the -n command-line option. The -r command-line option is used in conjunction with the #csv filter to produce the desired output.
Invocation
jq -n -R -r --slurpfile dict stream.json -f program.jq stream.txt
program.jq
(INDEX($dict[]; .uid) | map_values(.name)) as $d
| inputs
| split(",")
| map(fromjson)
| map($d[.])
| #csv
Caveat
The above assumes that the quoted values in stream.txt do not themselves contain commas.
If the quoted values in stream.txt do contain commas, then it would be much easier if the values given on each line in stream.txt were given as JSON entities, e.g. as an array of strings, or as a sequence of JSON strings with no separator character.
Solution to problem described in a comment
Invocation
< original.json jq -r --slurpfile dict stream.json -f program.jq
program.jq
(INDEX($dict[]; .uid) | map_values(.name)) as $d
| .source
| map($d[.])
| #csv
Struggling with formatting of data in jq. I have 2 issues.
Need to take the last array .rental_methods and concatenate them into 1 line, colon separated.
#csv doesn't seem to work with my query. I get the error string ("5343") cannot be csv-formatted, only array
jq command is this (without the | #csv)
jq --arg LOC "$LOC" '.last_updated as $lu | .data[]|.[]| $lu, .station_id, .name, .region_id, .address, .rental_methods[]'
JSON:
{
"last_updated": 1539122087,
"ttl": 60,
"data": {
"stations": [{
"station_id": "5343",
"name": "Lot",
"region_id": "461",
"address": "Austin",
"rental_methods": [
"KEY",
"APPLEPAY",
"ANDROIDPAY",
"TRANSITCARD",
"ACCOUNTNUMBER",
"PHONE"
]
}
]
}
}
I'd like the output to end up as:
1539122087,5343,Lot,461,Austin,KEY:APPLEPAY:ANDROIDPAY:TRANSITCARD:ACCOUNTNUMBER:PHONE:,
Using #csv:
jq -r '.last_updated as $lu
| .data[][]
| [$lu, .station_id, .name, .region_id, .address, (.rental_methods | join(":")) ]
| #csv'
What you were probably missing with #csv before was an array constructor around the list of things you wanted in the CSV record.
You could repair your jq filter as follows:
.last_updated as $lu
| .data[][]
| [$lu, .station_id, .name, .region_id, .address,
(.rental_methods | join(":"))]
| #csv
With your JSON, this would produce:
1539122087,"5343","Lot","461","Austin","KEY:APPLEPAY:ANDROIDPAY:TRANSITCARD:ACCOUNTNUMBER:PHONE"
... which is not quite what you've said you want. Changing the last line to:
map(tostring) | join(",")
results in:
1539122087,5343,Lot,461,Austin,KEY:APPLEPAY:ANDROIDPAY:TRANSITCARD:ACCOUNTNUMBER:PHONE
This is exactly what you've indicated you want except for the terminating punctuation, which you can easily add (e.g. by appending + "," to the program above) if so desired.
I'm trying to use jq to combine two arrays and running into a bit of trouble.
I'm attempting to parse out the data from netdata (netdata.firehol.org) and the two pieces of data within the json response that I"m interested in are both part of an array. The first array is labels for the datapoints in the second array.
Sample Input
[
"time",
"guest_nice",
"guest",
"steal",
"softirq",
"irq",
"user",
"system",
"nice",
"iowait"
]
[
1460728600,
0,
0,
0,
0.45731,
0,
0.25108,
11.74702,
48.22465,
0
]
Input
If you want to grab fresh data yourself to test against, you can use the following:
curl -s -X GET --header 'Accept: application/json'
'http://netdata.firehol.org/api/v1/data?chart=system.cpu&after=-10&before=0&points=1&group=average&format=json&options=seconds%2Cjsonwrap' | jq '.result.labels, .result.data[]'
I've tried to use map() as well as trying to assign vars to both arrays and then print out the objects together, but have been unsuccessful (below).
Code
| jq '.result.labels as $labels | .result.data[] as $data | .result.data[] | Label: $labels[.], data: $data[.]}'
I appreciate anyone's insight in advance as I'm a little stuck, and would prefer to be able to do this all in jq rather than using for loops in bash (if possible).
Expected Ouput
{
"time": "1460728600",
"guest_nice": "0",
...
}
You haven't specified exactly how you want the arrays to be combined, but one approach is to use transpose, which in this case is effectively a kind of zip. For example:
$ jq -n -c '[["a","b"], [1,2]] | transpose'
yields: [["a",1],["b",2]]
If you wanted an array of objects, then with the same input,
transpose | map( { (.[0]) : .[1] } )
would yield: [{"a":1},{"b":2}]
If your jq does not have transpose, here is its definition:
# transpose a possibly jagged matrix, quickly;
# rows are padded with nulls so the result is always rectangular.
def transpose:
[range(0; (map(length) | max)) as $j
| [range(0; length) as $i | .[$i][$j] ] ] ;
Alternatively, if you would prefer a very brief zip:
def zip: [range(0; .[0]|length) as $i | [.[0][$i], .[1][$i]]];
Here is a solution that handles the general case where the first array contains the key names and the following arrays contain values using transpose and from_entries
{h:.[0], v:.[1:][]} # {h:[keys], v:[values]}
| [.h, .v] # [ [keys], [values] ] ...
| [ transpose[] | {key:.[0], value:.[1]} ] # [ {"key":key, "value":value}, ... ]
| from_entries # { key:value, key:value, ... }
For example, if this filter is in filter.jq and data.json contains
["time","guest_nice","guest","steal","softirq","irq","user","system","nice","iowait"]
[1460728600,0,0,0,0.45731,0,0.25108,11.74702,48.22465,0]
[1460728601,0,0,0,0.45732,0,0.25109,12.74703,49,0]
then the command
jq -M -s -c -f filter.jq data.json
produces
{"time":1460728600,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45731,"irq":0,"user":0.25108,"system":11.74702,"nice":48.22465,"iowait":0}
{"time":1460728601,"guest_nice":0,"guest":0,"steal":0,"softirq":0.45732,"irq":0,"user":0.25109,"system":12.74703,"nice":49,"iowait":0}