Creating 1-1 mapping of Array results - arrays

I am trying to use data coming back from an ElasticSearch query, and get it into an Array in jsonata in a certain format.
Essentially, I need my result set to be like this:
{
"userName": [
"david2#david2.com",
"david2#david2.com",
"david2#david2.com",
"david2#david2.com"
],
"label": [
"Dealer",
"Inquiry",
"DD Test Skill1",
"_11DavidTest"
],
"value": [
3,
5,
2,
1
]
}
However, what I am getting is this:
{
"userName": "david2#david2.com",
"label": [
"Dealer",
"Inquiry",
"DD Test Skill1",
"_11DavidTest"
],
"value": [
3,
5,
2,
1
]
}
I am using the following to map the results:
(
$data := $map(data.hits.hits._source.item."Prod::User", function($v) {
{
"userName": $v.userName,
"label": $v.userSkillLevels.skill.name,
"value": $v.userSkillLevels.level
}
});
)
And my overall dataset returned form ElasticSearch is as follows:
{
"data": {
"took": 3,
"timed_out": false,
"_shards": {
"total": 15,
"successful": 15,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1.002851,
"hits": [
{
"_index": "items_latest_production1_user",
"_type": "_doc",
"_id": "63d000766f67d40a73073d5d_f6144acf2b3ff31209ef9f6d461cd849",
"_score": 1.002851,
"_source": {
"item": {
"Prod::User": {
"userSkillLevels": [
{
"level": 3,
"skill": {
"name": "Dealer"
}
},
{
"level": 5,
"skill": {
"name": "Inquiry"
}
},
{
"level": 2,
"skill": {
"name": "DD Test Skill1"
}
},
{
"level": 1,
"skill": {
"name": "_11DavidTest"
}
}
],
"userName": "david2#david2.com"
}
}
}
}
]
}
}
}
I can see that each user that comes back from Elastic, then as all the skills/levels in an array associated to 1 username.
I need to have the same number of userNames, as their are skills ... and am struggling to get it just right.
Thoughts? Appreciate any help, I'm sure I'm overlooking something simple.
Actually, need this format:
[
{
"userName" : "david2#david2.com"
"label" : "Dealer"
"value" : 3
},
{
"userName" : "david2#david2.com"
"label" : "Inquiry"
"value" : 5
},
{
"userName" : "david2#david2.com"
"label" : "DD Test Skill"
"value" : 2
},
{
"userName" : "david2#david2.com"
"label" : "_11DavidTest"
"value" : 1
}
]

One way you could solve this is to create the userName array yourself and making sure it has the same length as userSkillLevels.skill, here's how you can do this:
(
$data := $map(data.hits.hits._source.item."Prod::User", function($v, $i, $a) {
{
"userName": $map([1..$count($v.userSkillLevels.skill)], function() {$v.userName}),
"label": $v.userSkillLevels.skill.name,
"value": $v.userSkillLevels.level
}
});
)
Feel free to check out this response in Stedi JSONata Playground: https://stedi.link/PVwGacu

Actually, need this format:
For the updated format from your question, this solution can produce it using the parent operator:
data.hits.hits._source.item."Prod::User".userSkillLevels.{
"userName": %.userName,
"label": skill.name,
"value": level
}
Check it out on the playground: https://stedi.link/gexB3Cb

Related

Cypress intercept nested array and object and asserts it value

I am trying to assert value's in a requestbody which I have intercepted with cypress.
The values I need to assert are "name": "NewName", and "title": "STUB1-Draft" you can see in the requestbody example that I have attached.
My testscript in Cypress:
it.only('Check the requestbody', function () {
cy.intercept('PUT', '**/api/assessmenttest/**', req => {
req.reply({ statusCode: 200 });
}).as('NewSectionAndItem');
cy.wait('#NewSectionAndItem')
.its('request.body.test')
.its('testParts')
.its('testSections')
.its('name')
.should('include', 'NewName');
//cy.wait('#NewSectionAndItem').its('request.body.testParts').expect(arr_obj[1].name).to.equal('NewName')
The request body look like the following:
{
"structureStatistics": {
"testPartCount": 1,
"testSectionCount": 6,
"itemCount": 23
},
"name": "BIMMA",
"title": "OTAP",
"correctionInstructionsUrl": "C:Stub/FakePath/For/Cypress",
"correctionInstructionAppendices": [],
"testParts": [
{
"testSections": [
{
"order": 1,
"name": "Tekst 1 Looking for the one? ",
"itemReferences": [
{
"itemId": "55eb5a28-24d8-4705-b465-8e1454f73ac8",
"weight": 11,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "55eb5a28-24d8-4705-b465-8e1454f73ac8",
"title": "H-E-T1-1"
}
}
],
"id": "5c3eef2d-1094-4b9e-84c1-f184956f87fa"
},
{
"id": "ffaebc93-0bf6-4f75-944a-f61345a7be90",
"name": "NewName",
"itemReferences": [
{
"itemId": "58a29037-c92c-48f6-a7c3-a2f94e288992",
"weight": 0,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "58a29037-c92c-48f6-a7c3-a2f94e288992",
"title": "STUB1-Draft",
"state": "Draft"
}
}
]
},
{
"order": 2,
"name": "Tekst 2 The fruit Iron Ox bears",
"itemReferences": [
{
"itemId": "abfc0811-26c7-4d9d-b3cc-0c920e5af259",
"weight": 2,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "abfc0811-26c7-4d9d-b3cc-0c920e5af259",
"title": "H-E-T2-2"
}
},
{
"itemId": "3cfda5e0-0d64-44ef-8a4d-21f37484c024",
"weight": 12,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "3cfda5e0-0d64-44ef-8a4d-21f37484c024",
"title": "H-E-T2-3"
}
},
{
"itemId": "19ba8a53-9755-4beb-8f69-edd107b80230",
"weight": 1,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "19ba8a53-9755-4beb-8f69-edd107b80230",
"title": "H-E-T2-4"
}
},
{
"itemId": "3f5b7b81-df1f-4f01-8165-cb2226d9044d",
"weight": 1,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "3f5b7b81-df1f-4f01-8165-cb2226d9044d",
"title": "H-E-T2-5"
}
}
],
"id": "00f7455e-6d7d-4311-80cd-eff45c83ef2c"
},
{
"order": 3,
"name": "Tekst 3 How to live like a tramp",
"itemReferences": [
{
"itemId": "7e2d568c-4cde-4500-9c6b-c09f246155e4",
"weight": 1,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "7e2d568c-4cde-4500-9c6b-c09f246155e4",
"title": "H-E-T3-6"
}
},
{
"itemId": "87a5bf1c-451a-40b8-802a-53ee842cafcd",
"weight": 1,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "87a5bf1c-451a-40b8-802a-53ee842cafcd",
"title": "H-E-T3-7"
}
}
],
"id": "390ecc2e-6715-4898-aaea-158e790525a2"
}
],
"navigationMode": "Linear",
"submissionMode": "Individual",
"id": "a546a67c-ac39-4e81-bf03-beb482c920a0"
}
],
"metadataToBePublished": [
"be63002c-dcf8-449f-a0ae-6ba50d4e2712",
"4d70239e-7a6e-47c3-b157-462d6c8c5edc"
],
"created": "2022-07-08T09:00:00+00:00",
"modified": "2022-09-21T23:55:58.6532451+02:00",
"createdBy": {
"id": "a45ea6db-bf04-427d-9354-7081b7592a3d",
"fullName": "Manual Construction"
},
"lastModifiedBy": {
"id": "129a584c-a677-4d9f-b289-019d1815064f",
"fullName": "OZKAN"
},
"id": "300eea01-ee10-4bd9-9356-8aaa933e949c"
}
I could not figure out how I can assert nested arrays and value's, without using deep.equal for the complete request. Thank you indeed!
It basically just looks like a couple of things
an extra test property in the test that's not there in the request
testParts and testSections are arrays (square bracket instead of curly bracket), so you need an index for them
Generally I think this would work
cy.wait('#NewSectionAndItem')
.its('request.body')
.its('testParts.0')
.its('testSections.1')
.its('name')
.should('include', 'NewName');
The problem is identifying the correct array indexes. testParts has only one element, so 0 is the only index option there.
To find the testSection index, create a dummy .json file in VS Code (or other editor) paste in the json from above and use the collapse toggles on the left to easily see which index contains the value you seek.
You can use a cy-spok plugin to make a spok assertion to easily check a nested property.
Example use
const spok = require('cy-spok')
// later in your test
cy.wait('#NewSectionAndItem')
.its('request.body')
.should(spok({
test: {
testParts: {
testSections: {
name: 'NewName',
title: 'STUB1-Draft'
}
}
}
});

MongoDB : Update array in array if all objects of the array match multiple conditions

I want to update the array grades for a specific user. I want to push an object into grades if in the array there is no object that matches the semester and subject values.
Input :
{
"users": [
{
"userID": "id_1",
"grades": [
{
"semester": 1,
"subject": "math",
"value": 15
},
{
"semester": 1,
"subject": "french",
"value": 15
}
]
},
{
"userID": "id_2",
"grades": [
{
"semester": 1,
"subject": "math",
"value": 18
}
]
}
]
}
For example if I want to push :
{
"semester": 2,
"subject": "french",
"value": 16
}
for userID = id_1.
The result is :
{
"users": [
{
"userID": "id_1",
"grades": [
{
"semester": 1,
"subject": "math",
"value": 15
},
{
"semester": 1,
"subject": "french",
"value": 15
},
{
"semester": 2,
"subject": "french",
"value": 16
}
]
},
{
"userID": "id_2",
"grades": [
{
"semester": 1,
"subject": "math",
"value": 18
}
]
}
]
}
But also if I try to push
{
"semester": 1,
"subject": "french",
"value": 10
}
for userID = id_1.
It won't update, because there is already an object that match "semester" : 1 and "subject" : "french"
I tried to use arrayFilter with array identifier to filter on userID first, but then I cannot achieve to apply the push condition on grades array.
{"$push":{ "users.$[user].grades": { "semester": 1, "subject": "math", "value" : 10 } }}
arrayFilter = [{"user.userID" : "id_1"}]
Thank you in advance for the help.
You could use a mix of $not with $elemMatch to achieve what you're aiming to, which translate to if no element of grades match the condition on semester & subject of the new one, then add it,
Here's how you can do it:
db.users.update({
"userID": "id_1",
"grades": {
"$not": {
"$elemMatch": {
"semester": 2,
"subject": "french"
}
}
}
}, {
"$push": {
"grades": {
"semester": 2,
"subject": "french",
"value": 16
}
}
})

how to find duplicate array value on document mongodb

I got question might be many of you can help me.
so I have data on mongodb.
first data
{
"name" : 'david'
contacts : [
{
"name" : 'john',
"phone" : '123456'
},
{
"name" : 'george',
"phone" : '0987654'
}
]
}
second data
{
"name" : 'anita',
"contacts" : [
{
"name" : 'harry',
"phone" : '123456'
},
{
"name" : 'kurita',
"phone" : '323434'
}
]
}
the problem is,
can I query to find data that have duplicate contacts.phone.
so the result whill show like this.
{
"name" : 'david',
"contacts" : [
{
"name" : 'john',
"phone" : '123456'
}
]
}
{
"name" : 'anita',
"contacts" : [
{
"name" : 'harry',
"phone" : '123456'
}
]
}
data john and anita will show because they have similar data on contacts.phone
sorry for my english btw,
I hope you all understand what I mean.
thank you so much
There are a few steps involved to get the results you need.
We are going to write an aggregate pipeline to get the work done.
First you need to unwind your array values with the following:
{
$unwind: "$contacts"
}
Doc: https://docs.mongodb.com/manual/reference/operator/aggregation/unwind/
This would result into:
[
{
"_id": ObjectId("5a934e000102030405000000"),
"contacts": {
"name": "john",
"phone": "123456"
},
"name": "david"
},
{
"_id": ObjectId("5a934e000102030405000000"),
"contacts": {
"name": "george",
"phone": "0987654"
},
"name": "david"
},
{
"_id": ObjectId("5a934e000102030405000001"),
"contacts": {
"name": "harry",
"phone": "123456"
},
"name": "anita"
},
{
"_id": ObjectId("5a934e000102030405000001"),
"contacts": {
"name": "kurita",
"phone": "323434"
},
"name": "anita"
}
]
This would be much easier for us to group by field.
Doc: https://docs.mongodb.com/manual/reference/operator/aggregation/group/
{
$group: {
_id: {
phone: "$contacts.phone"
},
name: {
$addToSet: "$name"
},
contacts: {
$addToSet: "$contacts.name"
},
count: {
$sum: 1
}
}
}
That gives the following output:
[
{
"_id": {
"phone": "323434"
},
"contacts": [
"kurita"
],
"count": 1,
"name": [
"anita"
]
},
{
"_id": {
"phone": "123456"
},
"contacts": [
"john",
"harry"
],
"count": 2,
"name": [
"david",
"anita"
]
},
{
"_id": {
"phone": "0987654"
},
"contacts": [
"george"
],
"count": 1,
"name": [
"david"
]
}
]
Based on the output we need to match the count greater then 1 like:
Doc: https://docs.mongodb.com/manual/reference/operator/aggregation/match/
{
$match: {
count: {
"$gt": 1
}
}
}
Result is:
[
{
"_id": {
"phone": "123456"
},
"contacts": [
"john",
"harry"
],
"count": 2,
"name": [
"david",
"anita"
]
}
]
The query would look like:
Doc: https://docs.mongodb.com/manual/reference/operator/aggregation-pipeline/
db.collection.aggregate([
{
$unwind: "$contacts"
},
{
$group: {
_id: {
phone: "$contacts.phone"
},
name: {
$addToSet: "$name"
},
contacts: {
$addToSet: "$contacts.name"
},
count: {
$sum: 1
}
}
},
{
$match: {
count: {
"$gt": 1
}
}
}
])
MongoPlayground: https://mongoplayground.net/p/qSvhcYyAcQO
I hope this gives you a small idea what is possible with the aggregation pipeline.
Update / fix
According to your requirements you wish to have 2 objects foreach name that has duplicate contacts then you could use unwind again after the match.
[
{
"_id": {
"phone": "123456"
},
"contacts": [
"harry",
"john"
],
"count": 2,
"name": "david"
},
{
"_id": {
"phone": "123456"
},
"contacts": [
"harry",
"john"
],
"count": 2,
"name": "anita"
}
]
Cheers, Kevin

Elasticsearch terms aggregation by strings in an array

How can I write an Elasticsearch terms aggregation that splits the buckets by the entire term rather than individual tokens? For example, I would like to aggregate by state, but the following returns new, york, jersey and california as individual buckets, not New York and New Jersey and California as the buckets as expected:
curl -XPOST "http://localhost:9200/my_index/_search" -d'
{
"aggs" : {
"states" : {
"terms" : {
"field" : "states",
"size": 10
}
}
}
}'
My use case is like the one described here
https://www.elastic.co/guide/en/elasticsearch/guide/current/aggregations-and-analysis.html
with just one difference:
the city field is an array in my case.
Example object:
{
"states": ["New York", "New Jersey", "California"]
}
It seems that the proposed solution (mapping the field as not_analyzed) does not work for arrays.
My mapping:
{
"properties": {
"states": {
"type":"object",
"fields": {
"raw": {
"type":"object",
"index":"not_analyzed"
}
}
}
}
}
I have tried to replace "object" by "string" but this is not working either.
I think all you're missing is "states.raw" in your aggregation (note that, since no analyzer is specified, the "states" field is analyzed with the standard analyzer; the sub-field "raw" is "not_analyzed"). Though your mapping might bear looking at as well. When I tried your mapping against ES 2.0 I got some errors, but this worked:
PUT /test_index
{
"mappings": {
"doc": {
"properties": {
"states": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
Then I added a couple of docs:
POST /test_index/doc/_bulk
{"index":{"_id":1}}
{"states":["New York","New Jersey","California"]}
{"index":{"_id":2}}
{"states":["New York","North Carolina","North Dakota"]}
And this query seems to do what you want:
POST /test_index/_search
{
"size": 0,
"aggs" : {
"states" : {
"terms" : {
"field" : "states.raw",
"size": 10
}
}
}
}
returning:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0,
"hits": []
},
"aggregations": {
"states": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "New York",
"doc_count": 2
},
{
"key": "California",
"doc_count": 1
},
{
"key": "New Jersey",
"doc_count": 1
},
{
"key": "North Carolina",
"doc_count": 1
},
{
"key": "North Dakota",
"doc_count": 1
}
]
}
}
}
Here's the code I used to test it:
http://sense.qbox.io/gist/31851c3cfee8c1896eb4b53bc1ddd39ae87b173e

Array included in array search with elasticsearch

I have users indexed with categories as follows
{
id: 1
name: John
categories: [
{
id: 1
name: Category 1
},
{
id: 2
name: Category 2
}
]
},
{
id: 2
name: Mark
categories: [
{
id: 1
name: Category 1
}
]
}
And I'm trying to get all the documents with Category 1 or Category 2 with
{
filter:
{
bool: {
must: [
{
terms: {user.categories.id: [1, 2]}
}
]
}
}
}
But It only returns the first document that has the two categories, what I am doing wrong?
As I understood, terms search that one of the values is contained in the field, so for user 1
user.categories.id: [1, 2]
user 2
user.categories.id: [1]
Categoy id 1 is contained in both documents
The best way to handle this is probably with a nested filter. You'll have to specify the "nested" type in your mapping, though.
I can set up an index like this:
PUT /test_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"doc": {
"properties": {
"categories": {
"type": "nested",
"properties": {
"id": {
"type": "long"
},
"name": {
"type": "string"
}
}
},
"id": {
"type": "long"
},
"name": {
"type": "string"
}
}
}
}
}
then add some docs:
PUT /test_index/doc/1
{
"id": 1,
"name": "John",
"categories": [
{ "id": 1, "name": "Category 1" },
{ "id": 2, "name": "Category 2" }
]
}
PUT /test_index/doc/2
{
"id": 2,
"name": "Mark",
"categories": [
{ "id": 1, "name": "Category 1" }
]
}
PUT /test_index/doc/3
{
"id": 3,
"name": "Bill",
"categories": [
{ "id": 3, "name": "Category 3" },
{ "id": 4, "name": "Category 4" }
]
}
Now I can use a nested terms filter like this:
POST /test_index/doc/_search
{
"query": {
"constant_score": {
"filter": {
"nested": {
"path": "categories",
"filter": {
"terms": {
"categories.id": [1, 2]
}
}
}
},
"boost": 1.2
}
}
}
...
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "1",
"_score": 1,
"_source": {
"id": 1,
"name": "John",
"categories": [
{
"id": 1,
"name": "Category 1"
},
{
"id": 2,
"name": "Category 2"
}
]
}
},
{
"_index": "test_index",
"_type": "doc",
"_id": "2",
"_score": 1,
"_source": {
"id": 2,
"name": "Mark",
"categories": [
{
"id": 1,
"name": "Category 1"
}
]
}
}
]
}
}
Here is the code I used:
http://sense.qbox.io/gist/668aefe910643b52a3a10d40aca67104491668fc

Resources