I´m receiving this JSON as part of a SQL Query and I want to create nested objects out of the composed properties. This can be done easily if it was 1 element, but the MYSQL query returns an array right at the root:
[
{
"idlicense": 1,
"StartDate": "2022-11-15 00:00:00.0",
"EndDate": "2022-11-29 00:00:00.0",
"MonthlySearchMax": 500,
"Customer_CustomerId": 0,
"Customer_Guid": "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Customer_Name": "User",
"Customer_CustStartDate": "2022-11-15 00:00:00.0",
"Connector_ConnectorId": 0,
"Connector_Name": "connector0",
"Connector_Version": "1.01"
},
{
"idlicense": 2,
"StartDate": "2022-11-15 00:00:00.0",
"EndDate": "2022-11-29 00:00:00.0",
"MonthlySearchMax": 500,
"Customer_CustomerId": 0,
"Customer_Guid": "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Customer_Name": "User",
"Customer_CustStartDate": "2022-11-15 00:00:00.0",
"Connector_ConnectorId": 1,
"Connector_Name": "connector1",
"Connector_Version": "1.01"
}
]
I´m trying to create a nested JSON with JOLT on NIFI but can´t find the right format looking at the available examples and seems to be simple. I need my final JSON to look like this:
{
"Licenses": [
{
"idLicense": "1",
"Customer": {
"CustomerId": "0",
"Guid": "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Name": "User",
"CustStartDate": "2022-11-15 00:00:00.0"
},
"Connector": {
"ConnectorId": "0",
"Name": "Connector0",
"Version": "1.01"
}
},
{
"idLicense": "2",
"Customers": {
"CustomerId": "1",
"Guid": "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Name": "User",
"CustStartDate": "2022-11-15 00:00:00.0"
},
"Connector": {
"ConnectorId": "1",
"Name": "Connector1",
"Version": "1.01"
}
}
]
}
So far I´ve done this JOLT transform:
{
"idlicense": [1,2],
"Customer": {
"CustomerId": [0,0],
"Guid": ["c24c1fa3-0388-4c08-b431-8d0f05fe263a","c24c1fa3-0388-4c08-b431-8d0f05fe263a"],
"Name": ["User","User"],
"CustStartDate": ["2022-11-15 00:00:00.0","2022-11-15 00:00:00.0"]
},
"Connector": {
"ConnectorId": [0,1],
"Name": ["Conector0","Connector1"],
"Version": ["1.01","1.01"]
}
}
thank you for your support!
You can use this shift transformation spec
[
{
"operation": "shift",
"spec": {
"*": {
"idlicense": "Licenses[&1].&",
"*_*": "Licenses[&1].&(0,1).&(0,2)"
}
}
}
]
where &(0,1) represents before, &(0,2) after underscore character
Related
currently I´m receiving a JSON prefix soup Array and it make things much easier if I could transform it with JOLT to have nested objects and remove duplicates. So currently this is the JSON that is coming back from the source:
[
{
"License_idlicense": 1,
"License_StartDate": "2022-11-15 00:00:00.0",
"License_EndDate": "2022-11-29 00:00:00.0",
"License_MonthlySearchMax": 500,
"CustomerId": 0,
"Guid": "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Name": "Usuário Trial",
"CustStartDate": "2022-11-15 00:00:00.0",
"Connector_ConnectorId": 0,
"Connector_Name": "BigDataCorp",
"Connector_Version": "1.01"
},
{
"License_idlicense": 2,
"License_StartDate": "2022-11-15 00:00:00.0",
"License_EndDate": "2022-11-30 00:00:00.0",
"License_MonthlySearchMax": 500,
"CustomerId": 0,
"Guid": "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Name": "Usuário Trial",
"CustStartDate": "2022-11-15 00:00:00.0",
"Connector_ConnectorId": 1,
"Connector_Name": "Credilink",
"Connector_Version": "1.01"
}
]
and on another topic, #Barbaros helped me suggesting the following transform:
JOLT Transform from Prefix Soup to Nested
[
{
"operation": "shift",
"spec": {
"*": {
"CustomerId": "ECOLicense.Customers[&1].&",
"Guid": "ECOLicense.Customers[&1].&",
"Name": "ECOLicense.Customers[&1].&",
"*_*": "ECOLicense.Customers[&1].&(0,1).&(0,2)"
}
}
}
]
which results in the following:
{
"ECOLicense" : {
"Customers" : [ {
"CustomerId" : 0,
"Guid" : "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Name" : "Usuário Trial",
"License" : {
"idlicense" : 1,
"StartDate" : "2022-11-15 00:00:00.0",
"EndDate" : "2022-11-29 00:00:00.0",
"MonthlySearchMax" : 500
},
"Connector" : {
"ConnectorId" : 0,
"Name" : "BigDataCorp",
"Version" : "1.01"
}
}, {
"CustomerId" : 0,
"Guid" : "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Name" : "Usuário Trial",
"License" : {
"idlicense" : 2,
"StartDate" : "2022-11-15 00:00:00.0",
"EndDate" : "2022-11-30 00:00:00.0",
"MonthlySearchMax" : 500
},
"Connector" : {
"ConnectorId" : 1,
"Name" : "Credilink",
"Version" : "1.01"
}
} ]
}
}
Unfortunately, as you can see, the Main object Customers is duplicate but has different Connector and License objects. I would like to have the following result:
{
"ECOLicense": {
"Customers": [
{
"CustomerId": 0,
"Guid": "c24c1fa3-0388-4c08-b431-8d0f05fe263a",
"Name": "Usuário Trial",
"Licenses": [
{
"License": {
"idlicense": 1,
"StartDate": "2022-11-15 00:00:00.0",
"EndDate": "2022-11-29 00:00:00.0",
"MonthlySearchMax": 500
},
"Connector": {
"ConnectorId": 0,
"Name": "BigDataCorp",
"Version": "1.01"
}
},
{
"License": {
"idlicense": 2,
"StartDate": "2022-11-15 00:00:00.0",
"EndDate": "2022-11-30 00:00:00.0",
"MonthlySearchMax": 500
},
"Connector": {
"ConnectorId": 1,
"Name": "Credilink",
"Version": "1.01"
}
}
]
}
]
}
}
Seems to be a big challenge with several transformations. Thank ou for your support
You can add an extra node, Licenses[0] while converting Customers[&1] to Customers[0], and then apply a cardinality transformation spec such as
[
{
"operation": "shift",
"spec": {
"*": {
"CustomerId|Guid|Name": "ECOLicense.Customers[0].&",
"*_*": "ECOLicense.Customers[0].Licenses[&1].&(0,1).&(0,2)"
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"*": {
"*": "ONE",
"Licenses": "MANY"
}
}
}
}
}
]
I am trying to assert value's in a requestbody which I have intercepted with cypress.
The values I need to assert are "name": "NewName", and "title": "STUB1-Draft" you can see in the requestbody example that I have attached.
My testscript in Cypress:
it.only('Check the requestbody', function () {
cy.intercept('PUT', '**/api/assessmenttest/**', req => {
req.reply({ statusCode: 200 });
}).as('NewSectionAndItem');
cy.wait('#NewSectionAndItem')
.its('request.body.test')
.its('testParts')
.its('testSections')
.its('name')
.should('include', 'NewName');
//cy.wait('#NewSectionAndItem').its('request.body.testParts').expect(arr_obj[1].name).to.equal('NewName')
The request body look like the following:
{
"structureStatistics": {
"testPartCount": 1,
"testSectionCount": 6,
"itemCount": 23
},
"name": "BIMMA",
"title": "OTAP",
"correctionInstructionsUrl": "C:Stub/FakePath/For/Cypress",
"correctionInstructionAppendices": [],
"testParts": [
{
"testSections": [
{
"order": 1,
"name": "Tekst 1 Looking for the one? ",
"itemReferences": [
{
"itemId": "55eb5a28-24d8-4705-b465-8e1454f73ac8",
"weight": 11,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "55eb5a28-24d8-4705-b465-8e1454f73ac8",
"title": "H-E-T1-1"
}
}
],
"id": "5c3eef2d-1094-4b9e-84c1-f184956f87fa"
},
{
"id": "ffaebc93-0bf6-4f75-944a-f61345a7be90",
"name": "NewName",
"itemReferences": [
{
"itemId": "58a29037-c92c-48f6-a7c3-a2f94e288992",
"weight": 0,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "58a29037-c92c-48f6-a7c3-a2f94e288992",
"title": "STUB1-Draft",
"state": "Draft"
}
}
]
},
{
"order": 2,
"name": "Tekst 2 The fruit Iron Ox bears",
"itemReferences": [
{
"itemId": "abfc0811-26c7-4d9d-b3cc-0c920e5af259",
"weight": 2,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "abfc0811-26c7-4d9d-b3cc-0c920e5af259",
"title": "H-E-T2-2"
}
},
{
"itemId": "3cfda5e0-0d64-44ef-8a4d-21f37484c024",
"weight": 12,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "3cfda5e0-0d64-44ef-8a4d-21f37484c024",
"title": "H-E-T2-3"
}
},
{
"itemId": "19ba8a53-9755-4beb-8f69-edd107b80230",
"weight": 1,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "19ba8a53-9755-4beb-8f69-edd107b80230",
"title": "H-E-T2-4"
}
},
{
"itemId": "3f5b7b81-df1f-4f01-8165-cb2226d9044d",
"weight": 1,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "3f5b7b81-df1f-4f01-8165-cb2226d9044d",
"title": "H-E-T2-5"
}
}
],
"id": "00f7455e-6d7d-4311-80cd-eff45c83ef2c"
},
{
"order": 3,
"name": "Tekst 3 How to live like a tramp",
"itemReferences": [
{
"itemId": "7e2d568c-4cde-4500-9c6b-c09f246155e4",
"weight": 1,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "7e2d568c-4cde-4500-9c6b-c09f246155e4",
"title": "H-E-T3-6"
}
},
{
"itemId": "87a5bf1c-451a-40b8-802a-53ee842cafcd",
"weight": 1,
"neutralisationType": "NoNeutralisation",
"itemSummary": {
"id": "87a5bf1c-451a-40b8-802a-53ee842cafcd",
"title": "H-E-T3-7"
}
}
],
"id": "390ecc2e-6715-4898-aaea-158e790525a2"
}
],
"navigationMode": "Linear",
"submissionMode": "Individual",
"id": "a546a67c-ac39-4e81-bf03-beb482c920a0"
}
],
"metadataToBePublished": [
"be63002c-dcf8-449f-a0ae-6ba50d4e2712",
"4d70239e-7a6e-47c3-b157-462d6c8c5edc"
],
"created": "2022-07-08T09:00:00+00:00",
"modified": "2022-09-21T23:55:58.6532451+02:00",
"createdBy": {
"id": "a45ea6db-bf04-427d-9354-7081b7592a3d",
"fullName": "Manual Construction"
},
"lastModifiedBy": {
"id": "129a584c-a677-4d9f-b289-019d1815064f",
"fullName": "OZKAN"
},
"id": "300eea01-ee10-4bd9-9356-8aaa933e949c"
}
I could not figure out how I can assert nested arrays and value's, without using deep.equal for the complete request. Thank you indeed!
It basically just looks like a couple of things
an extra test property in the test that's not there in the request
testParts and testSections are arrays (square bracket instead of curly bracket), so you need an index for them
Generally I think this would work
cy.wait('#NewSectionAndItem')
.its('request.body')
.its('testParts.0')
.its('testSections.1')
.its('name')
.should('include', 'NewName');
The problem is identifying the correct array indexes. testParts has only one element, so 0 is the only index option there.
To find the testSection index, create a dummy .json file in VS Code (or other editor) paste in the json from above and use the collapse toggles on the left to easily see which index contains the value you seek.
You can use a cy-spok plugin to make a spok assertion to easily check a nested property.
Example use
const spok = require('cy-spok')
// later in your test
cy.wait('#NewSectionAndItem')
.its('request.body')
.should(spok({
test: {
testParts: {
testSections: {
name: 'NewName',
title: 'STUB1-Draft'
}
}
}
});
Using Elasticsearch I am trying to combine a nested array of objects into one array.
This is what my data looks like:
GET invoices/_search
{
"hits": [
{
"_index": "invoices",
"_id": "1234",
"_score": 1.0,
"_source": {
"id": 1234,
"status": "unpaid",
"total": 15.35,
"payments": [
{
"id": 1981,
"amount": 10,
"date": "2022-02-09T13:00:00+01:00"
},
{
"id": 1982,
"amount": 5.35,
"date": "2022-02-09T13:35:00+01:00"
}
]
}
},
# ... More hits
]
}
I want to only get the payments array of each hit combined into one array, so that it returns something like this:
{
"payments": [
{
"id": 1981,
"amount": 10,
"date": "2022-02-09T13:00:00+01:00"
},
{
"id": 1982,
"amount": 5.35,
"date": "2022-02-09T13:35:00+01:00"
},
{
"id": 5658,
"amount": 3,
"date": "2021-12-19T13:00:00+01:00"
}
]
}
I tried to get this result using nested queries but could not figure it out, the query I used is:
# Query I used:
GET invoices/_search
{
"_source": ["payments"],
"query": {
"nested": {
"path": "payments",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "payments.id"
}
}
]
}
}
}
}
}
# Result:
{
"hits": [
{
"_index": "invoices",
"_id": "545960",
"_score": 1.0,
"_source": {
"payments": [
{
"date": "2022-01-22T15:38:15+01:00",
"amount": 374.5,
"id": 320320
},
{
"date": "2022-01-22T15:30:03+01:00",
"amount": 160.5,
"id": 320316
}
]
}
},
{
"_index": "invoices",
"_id": "545961",
"_score": 1.0,
"_source": {
"payments": [
{
"date": "2022-01-22T15:38:15+01:00",
"amount": 12,
"id": 320350
},
{
"date": "2022-01-22T15:30:03+01:00",
"amount": 60.65,
"id": 320379
}
]
}
}
]
}
The result returns only the payments array but divided over multiple hits. How can I combine those arrays?
I'm currently working on a search using elasticsearch. We have a very large amount of users.
Here is the elasticsearch mapping:
PUT /example_index/_mapping/users
{
"properties": {
"user_autocomplete": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
},
"completion": {
"type": "text",
"analyzer": "user_autocomplete_analyzer",
"search_analyzer": "standard"
}
}
},
"firstName": {
"type": "text"
},
"lastName": {
"type": "text"
}
}
}
Here is the search query.
For example, I get 3 records
GET example_index/users/_search
{
"from": 0,
"size": 3,
"query": {
"query_string": {
"query": "*ro*",
"fields": [
"firstName",
"lastName"
]
}
},
"aggs": {
"user_suggestions": {
"terms": {
"size": 3,
"field": "user_autocomplete.raw"
}
}
}
}
Here is the output of elasticsearch
{
"took": 53,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 13,
"max_score": 1,
"hits": [
{
"_index": "example_index",
"_type": "users",
"_id": "08",
"_score": 1,
"_source": {
"firstName": "Eero",
"lastName": "Saarinen",
"user_autocomplete": "Eero Saarinen"
}
},
{
"_index": "example_index",
"_type": "users",
"_id": "16",
"_score": 1,
"_source": {
"firstName": "Aaron",
"lastName": "Judge",
"user_autocomplete": "Aaron Judge"
}
},
{
"_index": "example_index",
"_type": "users",
"_id": "20",
"_score": 1,
"_source": {
"firstName": "Robert",
"lastName": "Langdon",
"user_autocomplete": "Robert Langdon"
}
}
]
},
"aggregations": {
"user_suggestions": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 10,
"buckets": [
{
"key": "Eero Saarinen",
"doc_count": 1
},
{
"key": "Aaron Judge",
"doc_count": 1
},
{
"key": "Robert Langdon",
"doc_count": 1
}
]
}
}
}
I need result like in the following order:
Robert Langdon
Aaron Judge
Eero Saarinen
I have tried order method. It won't work. Is there a way to do this?
I need to aggregate an array as follows
Two document examples:
{
"_index": "log",
"_type": "travels",
"_id": "tnQsGy4lS0K6uT3Hwzzo-g",
"_score": 1,
"_source": {
"state": "saopaulo",
"date": "2014-10-30T17",
"traveler": "patrick",
"registry": "123123",
"cities": {
"saopaulo": 1,
"riodejaneiro": 2,
"total": 2
},
"reasons": [
"Entrega de encomenda"
],
"from": [
"CompraRapida"
]
}
},
{
"_index": "log",
"_type": "travels",
"_id": "tnQsGy4lS0K6uT3Hwzzo-g",
"_score": 1,
"_source": {
"state": "saopaulo",
"date": "2014-10-31T17",
"traveler": "patrick",
"registry": "123123",
"cities": {
"saopaulo": 1,
"curitiba": 1,
"total": 2
},
"reasons": [
"Entrega de encomenda"
],
"from": [
"CompraRapida"
]
}
},
I want to aggregate the cities array, to find out all the cities the traveler has gone to. I want something like this:
{
"traveler":{
"name":"patrick"
},
"cities":{
"saopaulo":2,
"riodejaneiro":2,
"curitiba":1,
"total":3
}
}
Where the total is the length of the cities array minus 1. I tried the terms aggregation and the sum, but couldn't output the desired output.
Changes in the document structure can be made, so if anything like that would help me, I'd be pleased to know.
in the document posted above "cities" is not a json array , it is a json object.
If changing the document structure is a possibility I would change cities in the document to be an array of object
example document:
cities : [
{
"name" :"saopaulo"
"visit_count" :"2",
},
{
"name" :"riodejaneiro"
"visit_count" :"1",
}
]
You would then need to set cities to be of type nested in the index mapping
"mappings": {
"<type_name>": {
"properties": {
"cities": {
"type": "nested",
"properties": {
"city": {
"type": "string"
},
"count": {
"type": "integer"
},
"value": {
"type": "long"
}
}
},
"date": {
"type": "date",
"format": "dateOptionalTime"
},
"registry": {
"type": "string"
},
"state": {
"type": "string"
},
"traveler": {
"type": "string"
}
}
}
}
After which you could use nested aggregation to get the city count per user.
The query would look something on these lines :
{
"query": {
"match": {
"traveler": "patrick"
}
},
"aggregations": {
"city_travelled": {
"nested": {
"path": "cities"
},
"aggs": {
"citycount": {
"cardinality": {
"field": "cities.city"
}
}
}
}
}
}