I'd like to take 50 or so json files, and simply concat them with jq.
Files look like this
file-1.json
{
"name": "john"
}
file-2.json
{
"name": "Xiaoming"
}
I want one file
file-all.json
That looks like:
[
{
"name": "Xiaoming"
},
{
"name": "Xiaoming"
}
]
An array of all the other files.
How do I do that? : (
If your files are named following a sequence like in your example, then you can use this:
jq -s '.' file-{1..50}.json > file-all.json
If you want all of the objects from those files combined in a single array:
jq -n '[inputs]' file-{1..50}.json > file-all.json
Related
This question already has answers here:
bash & jq: add attribute with object value
(2 answers)
Passing variable in jq to filter with select fails
(1 answer)
Closed 5 days ago.
I'm simply trying to replace an objects value in a json file with an array, using jq in a bash script.
The json file (truncated) looks like this:
{
"objects": {
"type": "foo",
"host": "1.1.1.1",
"port": "1234"
}
}
I want to replace the host objects value with an array of different values, so it looks like this:
{
"objects": {
"type": "foo",
"host": ["1.1.1.1","2.2.2.2"],
"port": "1234"
}
}
I tested around with this script. The Input comes from a simple, comma separated string which I convert into a proper json array (which seems to work).
But I'm not able replace the value with the array.
#!/bin/bash
objectshost="1.1.1.1,2.2.2.2"
objectshost_array=$(jq -c -n --arg arg $objectshost '$arg|split(",")')
jq --arg value "$objectshost_array" '.objects.host = $value' ./test.json > ./test.json.tmp
The best I ended up with, is this:
{
"objects": {
"type": "foo",
"host": "[\"1.1.1.1\",\"2.2.2.2\"]",
"port": "1234"
}
}
The result seems to be some logical result, as the script simply replaces the value with the arrays string. But it's not what I expected to get. ;)
I found some similar questions, but all of them were dealing with replacing values in existing arrays or key/value pairs, but my problem seems to be the conversion from a single value to an array.
Can somebody please push me into the right direction? Or should I forget about jq and threat the json file as a simple text file?
Thanks in advance,
André
It would work with a conditional assignment from arguments:
jq '
.objects.host = (
.objects.host |
if type == "array"
then .
else [ . ]
end + $ARGS.positional
)
' input.json --args 1.2.3.4 2.2.2.2 4.4.4.4
Or the same as a stand-alone jq script; which is more readable and maintainable:
myJQScript:
#!/usr/bin/env -S jq --from-file --args
.objects.host = (
.objects.host |
if type == "array"
then .
else [ . ]
end + $ARGS.positional
)
Make it executable:
chmod +x myJQScript
Run it with arguments to add array entries to host
$ ./myJQScript 1.2.3.4 2.2.2.2 4.4.4.4 < input.json
{
"objects": {
"type": "foo",
"host": [
"1.1.1.1",
"1.2.3.4",
"2.2.2.2",
"4.4.4.4"
],
"port": "1234"
}
}
You can do it with a single jq command:
#!/bin/sh
objectshost="1.1.1.1,2.2.2.2"
jq --arg value "$objectshost" '.objects.host = ($value / ",")' ./test.json > ./test.json.tmp
This has the added benefit of not requiring Bash arrays, but can be used with any shell.
If you already have a JSON array, you must use --argjson and not --arg. --arg always creates a variable of type string, --argjson however parses the value as JSON entity.
#!/bin/bash
objectshost="1.1.1.1,2.2.2.2"
objectshost_array=$(printf '%s\n' "$objectshost" | jq -c 'split(",")')
jq --argjson value "$objectshost_array" '.objects.host = $value' ./test.json > ./test.json.tmp
I have a series of JSON files that I've been working with via jq. Each file consists of an array of dictionaries, e.g.
file1.json: [{ "id": 1 }]
file2.json: [{ "id": 2 }]
The only command I have found which successfully merges all input files into one output array is:
jq --slurp '.[0] + .[1]' file1.json file2.json
This command outputs [{ "id": 1 }, { "id": 2 }], as expected.
I'm writing a shell script which is expected to merge a variable set of files into a single JSON array as an output. My script will execute something like:
find . -type f -iname '*.json' | xargs jq 'FILTER'
This should invoke jq like jq 'FILTER' file1.json file2.json ....
Is there a feature that I'm missing that will take all input files and first merge them into one contiguous list of objects without having to rewrite the filter to something like .[0] + .[1] + .[2] ...?
Given:
1.json
[{ "id": 1 }]
2.json
[{ "id": 2 }]
3.json
[{ "id": 3 }]
Then this command:
jq --slurp 'map(.[])' 1.json 2.json 3.json
Returns:
[
{
"id": 1
},
{
"id": 2
},
{
"id": 3
}
]
Or simply:
jq --slurp 'flatten' 1.json 2.json 3.json
It's generally best to avoid the -s option, especially if your version of jq supports inputs, as do all versions >= 1.5.
In any case, if your version of jq supports inputs, you could write:
jq -n '[inputs[]]' 1.json 2.json 3.json # etc
or whichever variant meets your needs.
Otherwise, you could simply write:
jq -s add 1.json 2.json 3.json # etc
Note on flatten
flatten itself is ruthless:
$ jq flatten <<< '[[[1], [[[[2]]]]]]'
[1,2]
flatten(1) is less so.
I want output in the below JSON array format with command on linux/bash platform . Can anybody help
data in text file
test:test
test1:test1
test4:test4
Expecting output:
{array :
[
{test:test},
{test1:test1},
{test4:test4}
]
}
Using jq command line JSON parser:
<file jq -Rs '{array:split("\n")|map(split(":")|{(.[0]):.[1]}?)}'
{
"array": [
{
"test": "test"
},
{
"test1": "test1"
},
{
"test4": "test4"
}
]
}
The options Rs let jq read the whole file as one string.
The script splits this string into pieces in order to have the expected format.
Assuming you want actual JSON output:
$ jq -nR '{array: (reduce inputs as $line ([]; . + [$line | split(":") | {(.[0]):.[1]}]))}' input.txt
{
"array": [
{
"test": "test"
},
{
"test1": "test1"
},
{
"test4": "test4"
}
]
}
Creating a JSON works best with a dedicated JSON tool that can also make sense of raw text.
xidel is such a tool.
XPath:
xidel -s input.txt -e '{"array":x:lines($raw) ! {substring-before(.,":"):substring-after(.,":")}}'
(x:lines($raw) is a shorthand for tokenize($raw,'\r\n?|\n'), which creates a sequence of all lines.)
XQuery:
xidel -s input.txt --xquery '{"array":for $x in x:lines($raw) let $a:=tokenize($x,":") return {$a[1]:$a[2]}}'
See also this Xidel online tester.
I try to rename the values in an array. However, only parts of them, keeping the other parts. Managed to rename whole strings, but not the "parts-task" using JQ.
JSON input:
{
"values": [
"foo:bar1",
"foo:bar2",
"foo:bar3"
]
}
desired output:
{
"values": [
"bar1",
"bar2",
"bar3"
]
}
Thank you in advance!
Assuming your jq has regex support (e.g. jq 1.5):
.values |= map(sub("foo:";"")))
Or maybe "^foo:"; ...
Input:
I have a filename called 'myseedips' with a set of Ip addresses in it in the below structure
10.204.99.15
10.204.99.12
10.204.99.41
These can be 'n' number of IP addressess line by line.
Output
I have no idea on bash programming. But I have to write a bash script to create a JSON file in the below structure. These IP addresses has to be in a loop, so that the JSON will change/extend depending on the length of myseedips file.
"cassandra": {
"nodes": [
{"ip_address": "10.204.99.15","type": "seed"},
{"ip_address": "10.204.99.12","type": "seed"},
{"ip_address": "10.204.99.41","type": "seed"}]
},
Also need to add logic to add comma at the end of each node for all nodes except the last. Do not append comma if there is only one node.
Example:
May be be something like the below code logic, but in bash programming.
j string
j = `"cassandra": {"nodes": [`
for i =0;i<len(ips);i++ {
j = j + `{"ip_address": "` + ips[i] + `","type": "seed"},`
}
j = j + `}]}`
Thanks
Nissar Sheik
Further to Jeff's answer, please note that the transformation can be accomplished with a single invocation of jq. If your jq has the inputs filter:
jq -Rn '[inputs] | {cassandra:{nodes:map({ip_address:.,type:"seed"})}}'
Otherwise:
jq -Rs 'split("\n") | {cassandra:{nodes:map({ip_address:.,type:"seed"})}}' ips.txt
Using jq, you'll need an extra pass to convert from raw text to a workable array but simple:
$ jq -R '.' myseedips | jq -s '{cassandra:{nodes:map({ip_address:.,type:"seed"})}}'
This yields the following:
{
"cassandra": {
"nodes": [
{
"ip_address": "10.204.99.15",
"type": "seed"
},
{
"ip_address": "10.204.99.12",
"type": "seed"
},
{
"ip_address": "10.204.99.41",
"type": "seed"
}
]
}
}
awk to the rescue!
a template awk solution can be
$ awk 'BEGIN{print "header"}
NR==FNR{c=NR;next}
{print "prefix",$1,"suffix" (FNR<c?",":"]")}
END{print "footer"}' myseedips{,}
header
prefix 10.204.99.15 suffix,
prefix 10.204.99.12 suffix,
prefix 10.204.99.41 suffix]
footer
you can replace the header,footer,prefix, and suffix.