I have a static dictionary which I need to combine (to add it) to dynamic dictionary.
"static": [
{
"item1": "abc",
"item2": "def",
"item3": "dkslks",
"item4": "kkk"
}
]
"dynamic": [
{
"item5": "aa",
"item6": "aaa",
"item7: "aaaa"
},
{
"item5": "bb",
"item6": "bbb",
"item7: "bbbb"
},
{
"item5": "cc",
"item6": "ccc",
"item7: "cccc"
}
]
I need to get the following result:
"combined": [
{
"item1": "abc",
"item2": "def",
"item3": "dkslks",
"item4": "kkk"
"item5": "aa",
"item6": "aaa",
"item7: "aaaa"
},
{
"item1": "abc",
"item2": "def",
"item3": "dkslks",
"item4": "kkk"
"item5": "bb",
"item6": "bbb",
"item7: "bbbb"
},
{
"item1": "abc",
"item2": "def",
"item3": "dkslks",
"item4": "kkk"
"item5": "cc",
"item6": "ccc",
"item7: "cccc"
}
]
I managed to combine it but just for the first part of this dict, not for all the members. Result was like this:
- set_fact:
combined: "{{ static|combine(dynamic) }}"
"combined": [
{
"item1": "abc",
"item2": "def",
"item3": "dkslks",
"item4": "kkk"
"item5": "aa",
"item6": "aaa",
"item7: "aaaa"
}
]
So I need to put it in a loop, but cannot find the right way. Here is some things I tried:
- set_fact:
combined: "{{ static | combine(json_query('[*]') | dynamic) }}"
or similar variations like this one
- set_fact:
combined: "{{ static | combine(dynamic) | json_query('[*]') }}"
or
- set_fact:
combined: "{{ dynamic | combine(static) }}"
loop: "{{dynamic | json_query('[*]')}}"
Would someone be kind to help?
Thanks!
A problem with several of your attempts is that both dynamic and static are lists, so they can't be used with the combine filter.
This seems to work with your sample data:
- hosts: localhost
gather_facts: false
vars_files:
- data.yaml
tasks:
- set_fact:
combined: "{{ combined + [item|combine(static[0])] }}"
vars:
combined: []
loop: "{{ dynamic }}"
- debug:
var: combined
This outputs:
PLAY [localhost] ***************************************************************
TASK [set_fact] ****************************************************************
ok: [localhost] => (item={'item5': 'aa', 'item6': 'aaa', 'item7': 'aaaa'})
ok: [localhost] => (item={'item5': 'bb', 'item6': 'bbb', 'item7': 'bbbb'})
ok: [localhost] => (item={'item5': 'cc', 'item6': 'ccc', 'item7': 'cccc'})
TASK [debug] *******************************************************************
ok: [localhost] => {
"combined": [
{
"item1": "abc",
"item2": "def",
"item3": "dkslks",
"item4": "kkk",
"item5": "aa",
"item6": "aaa",
"item7": "aaaa"
},
{
"item1": "abc",
"item2": "def",
"item3": "dkslks",
"item4": "kkk",
"item5": "bb",
"item6": "bbb",
"item7": "bbbb"
},
{
"item1": "abc",
"item2": "def",
"item3": "dkslks",
"item4": "kkk",
"item5": "cc",
"item6": "ccc",
"item7": "cccc"
}
]
}
PLAY RECAP *********************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Related
i want to compare collection with array in aggregate result
i have following two collection.
chat collection
chat.tags is a array value in reference key come from the tags collection.
"chat": [
{
"id": "test1",
"tags": [
"AAA",
"BBB",
"CCC",
"AAA"
]
},
{
"id": "test2",
"tags": [
"AAA",
"BBB",
"CCC"
]
}
]
tag collection
"tag": [
{
"id": "1234",
"key": "AAA",
"name": "a"
},
{
"id": "1235",
"key": "BBB",
"name": "b"
},
{
"id": "1236",
"key": "CCC",
"name": "c"
},
{
"id": "1237",
"key": "DDD",
"name": "d"
},
]
i want to result that id is "test1" and unique tags in chat collection.
i want to following result using with mongo aggregate.
Is it possible with from, let, pipeline when using lookup?
[
{
"chat": [
{
"id": "test1",
"setTags": [
"AAA",
"BBB",
"CCC"
]
}
],
"tag": [
{
"id": "1234",
"key": "AAA",
"name": "a"
},
{
"id": "1235",
"key": "BBB",
"name": "b"
},
{
"id": "1236",
"key": "CCC",
"name": "c"
}
]
}
]
please help me.
This can be achieved with a simple $lookup, like so:
db.chat.aggregate([
{
$match: {
id: "test1"
}
},
{
$lookup: {
from: "tag",
localField: "tags",
foreignField: "key",
as: "tagDocs"
}
},
{
$project: {
chat: [
{
id: "$id",
setTags: "$tags"
}
],
tag: "$tagDocs"
}
}
])
Mongo Playground
I didn't fully understand what the output structure you want is but it can easily be changed via a different $project stage.
--- EDIT ---
With Mongo's v3.6 $lookup syntax the pipeline remains the same, just the $lookup stage changes:
{
$lookup: {
from: "tag",
let: {
tagKeys: "$tags"
},
pipeline: [
{
$match: {
$expr: {
$in: [
"$key",
"$$tagKeys"
]
}
}
}
],
as: "tagDocs"
}
},
Mongo Playground
From this object:
{
"_id": null,
"reqs": [
["ab","ac","ab"],
["ac","cd"],
["ab","ae"]
]
}
How can I get this result using an aggregation:
{
"_id": null,
"reqs": ["ab","ac","cd","ae"]
}
Here it is:
mongos> db.a.find()
{ "_id" : ObjectId("5ff7a8c0bfa9f0b53f4b395d"), "reqs" : [ [ "ab", "ac", "ad" ], [ "ad", "dc", "ab" ], [ "ac", "ad", "dc" ] ] }
mongos> db.a.aggregate([{ $unwind:"$reqs"} , {$unwind:"$reqs" } , {$group:{_id:null , reqs:{$addToSet:"$reqs"}}} ])
{ "_id" : null, "reqs" : [ "ac", "ad", "dc", "ab" ] }
mongos>
I have the following JSON file.
[
{
"name": "first",
"Arrays": {
"dddd0001": [
"A",
"A",
"B",
"B",
"C",
"C",
"C",
"C",
"D",
"E",
"F"
]
}
},
{
"name": "second",
"Arrays": {
"dddd0002": [
"AA",
"AA",
"BA",
"BB",
"CC",
"CC",
"CC",
"CC",
"DD",
"DD",
"FF"
]
}
},
{
"name": "third",
"Arrays": {
"dddd0003": [
"1",
"1",
"2",
"3",
"3",
"4",
"4",
"4",
"0",
"0",
"0"
]
}
}
]
I need to remove duplicates inside every array in the JSON file. So the result should look like following
[
{
"name": "first",
"Arrays": {
"dddd0001": [
"A",
"B",
"C",
"D",
"E",
"F"
]
}
},
{
"name": "second",
"Arrays": {
"dddd0002": [
"AA",
"BA",
"BB",
"CC",
"DD",
"FF"
]
}
},
{
"name": "third",
"Arrays": {
"dddd0003": [
"1",
"2",
"3",
"4",
"0"
]
}
}
]
Array key names are not known in advance. There might be multiple arrays inside the Arrays object.
I tried to use unique_by but it requires the key name.
This algorithm - search for every array inside the Arrays object, for every such array apply unique function, re-assign results back to the array - should be fairly easy to implement, but I am stuck.
Thanks.
walk( if type == "array" then unique else . end)
If the original order should be respected, then you can easily use "def uniques" as defined at How do I get jq to return unique results when json has multiple identical entries?
you can use unique and |=:
$ jq '.[].Arrays[] |= unique' file.json
[
{
"name": "first",
"Arrays": {
"dddd0001": [
"A",
"B",
"C",
"D",
"E",
"F"
]
}
},
{
"name": "second",
"Arrays": {
"dddd0002": [
"AA",
"BA",
"BB",
"CC",
"DD",
"FF"
]
}
},
{
"name": "third",
"Arrays": {
"dddd0003": [
"0",
"1",
"2",
"3",
"4"
]
}
}
]
$
the only "problem" is that unique sorts the elements of the array, so for example contents of "dddd0003" array are not in the same order of your expected result. I don't know if this is could be a problem for you.
if "Arrays" property can also contain "non-array" values, extra care can be taken in order to "filter out" those "non-array" values so that unique doesn't complain.
select(type == "array") can be used: (output omitted):
$ jq '(.[].Arrays[] | select(type == "array")) |= unique' file.json
...
or arrays:
$ jq '(.[].Arrays[] | arrays) |= unique' file.json
...
these last two solutions better reflect your algorithm.
var jsonArr = [
{
"name": "first",
"Arrays": {
"dddd0001": [
"A",
"A",
"B",
"B",
"C",
"C",
"C",
"C",
"D",
"E",
"F"
]
}
},
{
"name": "second",
"Arrays": {
"dddd0002": [
"AA",
"AA",
"BA",
"BB",
"CC",
"CC",
"CC",
"CC",
"DD",
"DD",
"FF"
]
}
},
{
"name": "third",
"Arrays": {
"dddd0003": [
"1",
"1",
"2",
"3",
"3",
"4",
"4",
"4",
"0",
"0",
"0"
]
}
}
]
for(var i=0; i< jsonArr.length; i++)
{
var arrtemp = jsonArr[i][Object.keys(jsonArr[i])[1]];
var arrtmp2 = arrtemp[Object.keys(arrtemp)[0]];
jsonArr[i][Object.keys(jsonArr[i])[1]] =arrtmp2.filter((v, p) => arrtmp2.indexOf(v) == p);
console.log(jsonArr[i])
}
Let's say I have three documents in a collection, like so:
[
{"_id": "101", parts: ["a", "b"]},
{"_id": "102", parts: ["a", "c"]},
{"_id": "103", parts: ["a", "z"]},
]
what is the query I have to write so that if I input ["a","b","c"]
(i.e. all items in parts field value in each doc should be present in ["a","b","c"]) will output:
[
{"_id": "101", parts: ["a", "b"]},
{"_id": "102", parts: ["a", "c"]}
]
is this even possible? any idea?
Below solution may not be the best but it works. The idea is finding all documents that has no items in parts outside the input array. It can be done with combination of $not, $elemMatch and $nin:
db.collection.find({
parts: {
$not: {
"$elemMatch": {
$nin: ["a", "b", "c"]
}
}
}
})
Mongo Playground
Thanks to #prasad_. I have tried to come up with a solution which is similar to what I wanted. I have used $setDifference here.
db.collection.aggregate([
{
$project: {
diff: {
$setDifference: [
"$parts",
[
"a",
"b",
"c"
]
]
},
document: "$$ROOT"
}
},
{
$match: {
"diff": {
$eq: []
}
}
},
{
$project: {
"diff": 0
}
},
])
output:
[
{
"_id": "101",
"document": {
"_id": "101",
"parts": [
"a",
"b"
]
}
},
{
"_id": "102",
"document": {
"_id": "102",
"parts": [
"a",
"c"
]
}
}
]
Mongo Playground
I would like to group a JSON array by two properties (name and address2) and then reduce it. Here is the current JSON:
{
"clientId": "FOO",
"name": "John Doe",
"address1": "London",
"address2": "1 High Street",
"invoiceNumber": "1234",
"itemDescription": "Item 1",
"itemQuantity": 1
},
{
"clientId": "BAR",
"name": "John Doe",
"address1": "London",
"address2": "1 Oxford Street",
"invoiceNumber": "0987",
"itemDescription": "Item 2",
"itemQuantity": 1
},
{
"clientId": "FOO",
"name": "John Doe",
"address1": "London",
"address2": "1 High Street",
"invoiceNumber": "5678",
"itemDescription": "Item 3",
"itemQuantity": 1
}
This is what I would like to achieve:
[
{
"clientId": "FOO",
"name": "John Doe",
"address1": "London",
"address2": "1 High Street",
"deliveries": [
{
"invoiceNumber": "1234",
"itemQuantity": 1,
"itemDescription": "Item 1"
},
{
"invoiceNumber": "5678",
"itemDescription": "Item3",
"itemQuantity": 1
}
]
},
{
"clientId": "BAR",
"name": "JohnDoe",
"address1": "London",
"address2": "12 Oxford Street",
"deliveries": [
{
"invoiceNumber": "0987",
"itemDescription": "Item2",
"itemQuantity": 1
}
]
}
]
And here is the code that I have so far, which doesn't produce what I would hope it would:
var groups = _.groupBy(rows, function(value) {
return value.name + '#' + value.address2;
});
var data = _.map(groups, function(group) {
return {
clientId: group[0].clientId,
name: group[0].name,
address1: group[0].address1,
address2: group[0].address2,
deliveries: [{
invoiceNumber: _.map(group,'invoiceNumber'),
itemDescription: _.map(group,'itemDescription'),
itemQuantity: _.map(group,'itemQuantity'),
}]
}
});
And it produces the following output:
[ { clientId: 'FOO',
name: 'John Doe',
address1: 'London',
address2: '1 High Street',
deliveries: [ [Object] ] } ]
I am not sure how to create an array of "deliveries" objects within the lodash _.map function. Please help! :)
Use one map that picks out the properties you want from a group:
...
deliveries: _.map(group, item => _.pick(item, ['invoiceNumber', 'itemDescription', 'itemQuantity' ]))
...
You weren't far off!
Had a go at making it a little bit more succinct by using ES6 syntax and incorporating _.pick() into your code, as suggested by Gruff.
If you really like one-liners, that's possible, too! Lodash makes it easy to chain/nest functions to do this, although you may compromise some readability:
// Data:
const rows = [{
"clientId": "FOO",
"name": "John Doe",
"address1": "London",
"address2": "1 High Street",
"invoiceNumber": "1234",
"itemDescription": "Item 1",
"itemQuantity": 1
},
{
"clientId": "BAR",
"name": "John Doe",
"address1": "London",
"address2": "1 Oxford Street",
"invoiceNumber": "0987",
"itemDescription": "Item 2",
"itemQuantity": 1
},
{
"clientId": "FOO",
"name": "John Doe",
"address1": "London",
"address2": "1 High Street",
"invoiceNumber": "5678",
"itemDescription": "Item 3",
"itemQuantity": 1
}];
// Result:
const groups = _.groupBy(rows, item => item.clientId);
const result = _.map(groups, group => {
const {clientId, name, address1, address2} = group[0];
const deliveries = _.map(group, item => _.pick(item, ['invoiceNumber', 'itemDescription', 'itemQuantity']));
return {clientId, name, address1, address2, deliveries};
});
console.log('multi-liner result: ', result);
// One-line version:
const oneLiner = _.map(_.groupBy(rows, item => item.clientId), group => _.merge(_.pick(group[0], ['clientId', 'name', 'address1', 'address2']), {deliveries: _.map(group, item => _.pick(item, ['invoiceNumber', 'itemDescription', 'itemQuantity']))}));
console.log('one-liner result: ', oneLiner);
<script src="https://cdn.jsdelivr.net/lodash/4.17.1/lodash.min.js"></script>
Hope that helps! If you'd like more info on the Lodash functions used, here's the official documentation for them:
_.groupBy()
_.map()
_.pick()