I have the following returned to me as a response of a mocking tool I'm using.
{
"mappings" : [
{
"id" : "bcf3559f-7ff7-406b-a4f1-6d3e9ac00e63",
"name" : "Hellow world 2",
"request" : {
"url" : "/hello-world-2",
"method" : "POST"
},
"response" : {
"status" : 200,
"body" : "\nBody content for stub 3\n\n",
"headers" : { }
},
"uuid" : "bcf3559f-7ff7-406b-a4f1-6d3e9ac00e63",
"persistent" : true,
"priority" : 5
},
{
"id" : "9086b24f-4f5e-465a-bbe5-73bbfb82cd5c",
"name": "Hello world",
"request" : {
"url" : "/hello-world",
"method" : "ANY"
},
"response" : {
"status" : 200,
"body" : "Hi!"
},
"uuid" : "9086b24f-4f5e-465a-bbe5-73bbfb82cd5c"
} ]
}
I'd like to know how I can split each object into it's own file with the file named after the id of the object.
E.g:
bcf3559f-7ff7-406b-a4f1-6d3e9ac00e63.json
bcf3559f-7ff7-406b-a4f1-6d3e9ac00e63.json
I have got as far as this so far but can't get it over the line:
jq -c '.mappings = (.mappings[] | [.])' mappings.json |
while read -r json ; do
N=$((N+1))
jq . <<< "$json" > "tmp/file${N}.json"
done
I'd recommend printing the id on one line, and the corresponding object on the next. For example:
jq -c '.mappings[] | .id, .' mappings.json |
while read -r id ; do
echo "id=$id"
read -r json
jq . <<< "$json" > "tmp/${id}.json"
done
I would write a simple Python script instead (or the equivalent in your favorite, general-purpose programming language).
import sys, json
d = json.load(sys.stdin):
for o in d['mappings']:
with open(os.path.join('tmp', o['id'] + '.json'), 'w') as f:
json.dump(o, f)
This would be more efficient and less error-prone, at least until jq gets some sort of output built-in:
# hypothetical
jq '.mappings[] | output("tmp/\(.id).json")' mappings.json
Related
My dedicated servers are generating 2 laptime arrays and I would like to use a script to merge them into a single, new json file, with duplicate "steamids" removed (and kept grouped together as they still are) and both arrays under a single loggedTimes {} (so I can feed it to a html script that produces laptimes and a leaderboard). In other words, I want the structure to remain.
The first laptime file and the second laptime file go through the following command
jq 'reduce . as $item ({}; . * $item)' laptimes_data_ams.json laptimes_data_kow.json > laptimes.json
to then generate the (badly) merged laptime file.
I can get a file reduced but can't get any further than that. I checked threads by other around here and whenever I try their suggestions the script just refuses to work. Anybody available to lend me a hand in generating a working script to keep this final structure post-merge?
{
"loggedTimes" : {
steamids" : {
"idnumber1" : "name1",
"idnumber2" : "name2"
},
"vehicles" : {
"vehiclenumber1" : {
"laptimes" : {
"idnumber1" : {
"lapTime" : time1,
"logtime" : log1,
"name" : "name 1",
"rank" : rank1,
"refId" : id1,
"vehicleid" : vehiclenumber1,
"wet" : 0
},
"idnumber2" : {
"lapTime" : time2,
"logtime" : log2,
"name" : "name 2",
"rank" : rank2,
"refId" : id2,
"vehicleid" : vehiclenumber1,
"wet" : 0
}
}
}
"vehiclesnumber2" : {
//you get the idea by now
}
}
}
You haven't specified how the merge is to be performed, but one option would be to let the key-value pairs in the second file dominate. In that case, you could write:
jq -n '
input as $one
| input as $two
| ($one + $two)
| .loggedTimes.steamids = ($one.loggedTimes.steamids + $two.loggedTimes.steamids)
' 1.json 2.json
With your input, this produces output from which the following is an extract:
{
"loggedTimes": {
"steamids": {
"76561197960277005": "[DECOCO]koker_SZ",
"76561197960436395": "JOJO",
...
},
"vehicles": {
"-1142039519": {
"lapTimes": {}
},
"-1201605905": {
"lapTimes": {
"76561197984026143": {
"lapTime": 609101,
"logtime": 1606516985,
"name": "Night Dick",
"rank": 1,
"refId": 36032,
"vehicleId": -1201605905,
"wet": 0
}
}
}
...
}
}
}
In Python 3-x, consider you have an array in JSON syntax:
members = '''[
{
"name" : "Amber",
"age" : 5
},
{
"name" : "Becky",
"age" : 4
}
]'''
How do you get the value for age where the name is Amber? (The answer should be 5).
variable members look like a string so first change string to json object and search what you want.
members = '''[
{
"name" : "Amber",
"age" : 5
},
{
"name" : "Becky",
"age" : 4
}
]'''
import json
obj = json.loads(members) #Changing string to json
for some_variable in obj:
if some_variable['name'] == 'Amber':
print (some_variable['age']) # will print 5
This question already has answers here:
Escape field name in jq that contains '#' and '-'? [duplicate]
(2 answers)
Closed 3 years ago.
I have a large dump of data in a file.json that looks like:
[{
"recordList" : {
"record" : [{
"Production" : {
"creator.role" : {
"term" : "A"
}
}
},
{
"Production" : {}
},
{
"Production" : {
"creator.role" : {
"term" : ""
}
}
},
{
"Production" : [
{
"creator.role" : {"term" : "B"}
},
{
"creator.role" : {"term" : ""}
}
]
}]
}
}]
I need to check if there is at least one 'term' (that is not empty) for 'creator.role' in a record or not. If there is I give a 1 else a 0 for that field in a CSV-file.
Thanks to the answers on an earlier post, I managed to access a field 'creator' although it could be an object or an array (see: Accessing field with jq that can be string or array).
But now I also have the same problem for the field 'creator.role' with the special character '.' and don't know how to handle that.
The code I tried:
jq -r '.[].recordList.record[].Production | "\(if ((type == "array" and .[0]["creator.role"].term and .[0]["creator.role"].term !="") or (type == "object" and ["creator.role"].term and ["creator.role"].term !="")) then 1 else 0 end),"' file.json
I get this Error:
Cannot index array with string "term"
The output I want to get in this case is:
1,
0,
0,
1,
jq solution:
jq -r '.[].recordList.record[].Production
| "\(if ((type == "array" and any(.["creator.role"].term !=""))
or (type == "object" and .["creator.role"].term and .["creator.role"].term !=""))
then 1 else 0 end),"' file.json
I am new to ES and but I'm getting the hang of it.
It's a really powerful piece of software, but I have to say that the documentation is really lacking and confusing some times.
Here's my question:
I have an integer array, that looks like this:
"hits_history" : [0,0]
I want to append an integer to that array via an "update_by_query" call, I searched and found this link: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html
which has this example:
POST test/type1/1/_update
{
"script" : {
"inline": "ctx._source.tags.add(params.tag)",
"lang": "painless",
"params" : {
"tag" : "blue"
}
}
}
so I tried:
curl -XPOST 'localhost:9200/example/example/_update_by_query?pretty' -H 'Content-Type: application/json' -d'
{
"script": {
"inline": "ctx._source.hits_history.add(params.hits)",
"params": {"hits": 0}
},
"query": {
"match_all": {}
}
}
'
but it gave me this error:
"ctx._source.hits_history.add(params.hits); ",
" ^---- HERE"
"type" : "script_exception",
"reason" : "runtime error",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Unable to find dynamic method [add] with [1] arguments for class [java.lang.Integer]."
So, I looked further and found this: https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html
which has this example:
We can also use a script to add a new tag to the tags array.
POST /website/blog/1/_update
{
"script" : "ctx._source.tags+=new_tag",
"params" : {
"new_tag" : "search"
}
}
So I tried it:
curl -XPOST 'localhost:9200/example/example/_update_by_query?pretty' -H 'Content-Type: application/json' -d'
{
"script": {
"inline": "ctx._source.hits_history += 0;"
},
"query": {
"match_all": {}
}
}
'
Result:
"type" : "script_exception",
"reason" : "runtime error",
"caused_by" : {
"type" : "class_cast_exception",
"reason" : "Cannot apply [+] operation to types [java.util.ArrayList] and [java.lang.Integer]."
So, how can I append items to the arrayList? Is there a more up-to-date documentation I should look into?
What I wanted to do was simply something like this:
ctx._source.hits_history.add(ctx._source.today_hits);
ctx._source.today_hits = 0;
Thank you
You should store first value as array (containing one value).
Then you can use add() method.
POST /website/blog/1/_update
{
"script" : "if (ctx._source.containsKey('tags')) { ctx._source.tags.add('next') } else { ctx._source.tags = ['first'] }"
}
I'm using MongoDB shell version: 2.4.8, and would simply like to know why a nested array search doesn't work quite as expected.
Assume we have 2 document collections, (a) Users:
{
"_id" : ObjectId("u1"),
"username" : "user1",
"org_ids" : [
ObjectId("o1"),
ObjectId("o2")
]
}
{
"_id" : ObjectId("u2"),
"username" : "user2",
"org_ids" : [
ObjectId("o1")
]
}
and (b) Organisations:
{
"_id" : ObjectId("o1"),
"name" : "Org 1"
}
{
"_id" : "ObjectId("o2"),
"name" : "Org 2"
}
Collections have indexes defined for
Users._id, Users.org_id, Organisations._id
I would like to find all Organisations a specific user is a member of.
I've tried this:
> myUser = db.Users.find( { _id: ObjectId("u1") })
> db.Organisations.find( { _id : { $in : [myUser.org_ids] }})
yet it yields nothing as a result. I've also tried this:
> myUser = db.Users.find( { _id: ObjectId("u1") })
> db.Organisations.find( { _id : { $in : myUser.org_ids }})
but it outputs the error:
error: { "$err" : "invalid query", "code" : 12580 }
(which basically says you need to pass $in an array) ... but that's what I thought I was doing originally ? baffled.
Any ideas what I'm doing wrong?
db.collection.find() returns a cursor - according to documentation. Then myUser.org_ids is undefined, but $in field must be an array. Let's see the solution!
_id is unique in a collection. So you can do findOne:
myUser = db.Users.findOne( { _id: ObjectId("u1") })
db.Organisations.find( { _id : { $in : myUser.org_ids }})
If you are searching for a non-unique field you can use toArray:
myUsers = db.Users.find( { username: /^user/ }).toArray()
Then myUsers will be an array of objects matching to the query.