I want to remove a name/value pair from inside a json array and print it outside. I started by trying this and then expanding the whole request to be a json array. The solution mentioned above does not seem to be working.
Input :
[
{
"createdBy": "Admin",
"createdDate": "2022-09-08",
"modifiedBy": "Admin",
"attrs": [
{
"name": "Type",
"value": "Postpaid"
},
{
"name": "subscriber",
"value": "Paid"
},
{
"name": "Details",
"value": {
"createdDate": "today",
"description": "offer",
"id": null
}
}
],
"relatedInfo": [
{
"type": "Number",
"name": "000000"
},
{
"type": "Type",
"name": "Post"
}
]
},
{
"createdBy": "Admin",
"createdDate": "2022-09-08",
"modifiedBy": "Admin",
"attrs": [
{
"name": "Type",
"value": "Postpaid"
},
{
"name": "subscriber",
"value": "Paid"
},
{
"name": "Details",
"value": {
"createdDate": "today",
"description": "offer",
"id": null
}
}
],
"relatedInfo": [
{
"type": "Number",
"name": "000000"
},
{
"type": "Type",
"name": "Post"
}
]
}
]
Desired Output :
[
{
"createdBy": "Admin",
"createdDate": "2022-09-08",
"modifiedBy": "Admin",
"attrs": [
{
"name": "Type",
"value": "Postpaid"
},
{
"name": "subscriber",
"value": "Paid"
}
],
"Details": {
"createdDate": "today",
"description": "offer",
"id": null
},
"relatedInfo": [
{
"type": "Number",
"name": "000000"
},
{
"type": "Type",
"name": "Post"
}
]
},
{
"createdBy": "Admin",
"createdDate": "2022-09-08",
"modifiedBy": "Admin",
"attrs": [
{
"name": "Type",
"value": "Postpaid"
},
{
"name": "subscriber",
"value": "Paid"
}
],
"Details": {
"createdDate": "today",
"description": "offer",
"id": null
},
"relatedInfo": [
{
"type": "Number",
"name": "000000"
},
{
"type": "Type",
"name": "Post"
}
]
}
]
Current Jolt spec:
[
{
"operation": "shift",
"spec": {
"*": "[&]",
"attrs": {
"*": {
"name": {
"*": { "#2": "&4" },
"Details": {
"#(2,value)": "&1"
}
}
}
}
}
}
]
I can't seem to figure out how the jolt spec would change in case of the array
So far so good, just need to combine the attributes at a common node. To do this, I've used the identifiers [&1] and [&5] in order to reach the level of the outermost index within the tree such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": "[&1].&",
"attrs": {
"*": {
"name": {
"*": {
"#2": "[&5].&4"
},
"Details": {
"#(2,value)": "[&5].&1"
}
}
}
}
}
}
}
]
Related
What I would like to do it if there is addOn array, I would like to append the word "xxx" to the end of the name.
Schema applied to message.
{
"properties": {
"appointment": {
"properties": {
"id": {
"type": "integer"
},
"lines": {
"items": {
"properties": {
"addOn": {
"items": {
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "string"
}
},
"required": [
"id",
"name"
],
"type": "object"
},
"type": "array"
},
"id": {
"type": "integer"
},
"price": {
"type": "integer"
}
},
"required": [
"id",
"price"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
},
"messageId": {
"type": "string"
}
},
"type": "object"
}
Message 1
{
"messageId": "11",
"appointment": {
"id": 22,
"lines": [
{
"id": 33,
"price": 125.0,
"addOn": [
{
"id": 44,
"name": "test"
}
]
}
]
}
}
Message 2
{
"messageId": "11",
"appointment": {
"id": 22,
"lines": [
{
"id": 33,
"price": 125.0
}
]
}
}
Message 1 works fine but whenever I try and use length or Parse Array I get a message that addOn is null.
How can I put a proper Condition express to not get any errors or do nothing when there is no addOn array.
Logic App.
Message1 - Okay
Message2 - Error
Code View.
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"For_each": {
"actions": {
"For_each_2": {
"actions": {
"Condition_3": {
"actions": {},
"expression": {
"and": [
{
"equals": [
"#empty(items('For_each_2'))",
"True"
]
}
]
},
"runAfter": {},
"type": "If"
}
},
"foreach": "#items('For_each')['addOn']",
"runAfter": {},
"type": "Foreach"
}
},
"foreach": "#body('Parse_JSON')?['appointment']?['lines']",
"runAfter": {
"Parse_JSON": [
"Succeeded"
]
},
"type": "Foreach"
},
"Parse_JSON": {
"inputs": {
"content": "#triggerBody()",
"schema": {
"properties": {
"appointment": {
"properties": {
"id": {
"type": "integer"
},
"lines": {
"items": {
"properties": {
"addOn": {
"items": {
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "string"
}
},
"required": [
"id",
"name"
],
"type": "object"
},
"type": "array"
},
"id": {
"type": "integer"
},
"price": {
"type": "integer"
}
},
"required": [
"id",
"price"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
},
"messageId": {
"type": "string"
}
},
"type": "object"
}
},
"runAfter": {},
"type": "ParseJson"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {},
"triggers": {
"manual": {
"inputs": {
"schema": {
"properties": {
"appointment": {
"properties": {
"id": {
"type": "integer"
},
"lines": {
"items": {
"properties": {
"addOn": {
"items": {
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "string"
}
},
"required": [
"id",
"name"
],
"type": "object"
},
"type": "array"
},
"id": {
"type": "integer"
},
"price": {
"type": "integer"
}
},
"required": [
"id",
"price",
"addOn"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
},
"messageId": {
"type": "string"
}
},
"type": "object"
}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
Steve
I would like to append the word "xxx" to the end of the name.
For appending you can directly add while setting the "name" variable.
Alternatively, you can also use concat function as below.
concat(body('Parse_JSON')?['appointment']?['lines'][0]['addOn'][0]['name'],variables('xxx'))
where both cases would give the result as the following
How can I put a proper Condition express to not get any errors or do nothing when there is no addOn array.
You can do this in many ways where one of the workarounds would be using if the lines contain "addOn" or not.
"contains": ["#body('Parse_JSON')?['appointment']?['lines'][0]","addOn"]
MESSAGE1 RESULTS:
MESSAGE2 RESULTS:
You can reproduce the same in your logic app using the below codeview
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Compose": {
"inputs": "#variables('Message')",
"runAfter": {
"For_each": [
"Succeeded"
]
},
"type": "Compose"
},
"For_each": {
"actions": {
"Condition": {
"actions": {
"Set_variable": {
"inputs": {
"name": "Message",
"value": {
"appointment": {
"id": "#body('Parse_JSON')?['appointment']?['id']",
"lines": [
{
"addOn": "addOn is NULL",
"id": "#items('For_each')?['id']",
"price": "#items('For_each')?['price']"
}
]
},
"messageId": "#{body('Parse_JSON')?['messageId']}"
}
},
"runAfter": {},
"type": "SetVariable"
}
},
"else": {
"actions": {
"For_each_2": {
"actions": {
"Set_variable_2": {
"inputs": {
"name": "Message",
"value": {
"appointment": {
"id": "#body('Parse_JSON')?['appointment']?['id']",
"lines": [
{
"addOn": [
{
"id": "#items('For_each_2')?['id']",
"name": "#{concat(body('Parse_JSON')?['appointment']?['lines'][0]['addOn'][0]['name'],variables('xxx'))}"
}
],
"id": "#items('For_each')?['id']",
"price": "#items('For_each')?['price']"
}
]
},
"messageId": "#{body('Parse_JSON')?['messageId']}"
}
},
"runAfter": {},
"type": "SetVariable"
}
},
"foreach": "#items('For_each')['addOn']",
"runAfter": {},
"type": "Foreach"
}
}
},
"expression": {
"and": [
{
"not": {
"contains": [
"#body('Parse_JSON')?['appointment']?['lines'][0]",
"addOn"
]
}
}
]
},
"runAfter": {},
"type": "If"
}
},
"foreach": "#body('Parse_JSON')?['appointment']?['lines']",
"runAfter": {
"Initialize_variable_xxx": [
"Succeeded"
]
},
"type": "Foreach"
},
"Initialize_variable": {
"inputs": {
"variables": [
{
"name": "Message",
"type": "object"
}
]
},
"runAfter": {
"Parse_JSON": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Initialize_variable_xxx": {
"inputs": {
"variables": [
{
"name": "xxx",
"type": "string",
"value": "xxx"
}
]
},
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Parse_JSON": {
"inputs": {
"content": "#triggerBody()",
"schema": {
"properties": {
"appointment": {
"properties": {
"id": {
"type": "integer"
},
"lines": {
"items": {
"properties": {
"addOn": {
"items": {
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "string"
}
},
"required": [
"id",
"name"
],
"type": "object"
},
"type": "array"
},
"id": {
"type": "integer"
},
"price": {
"type": "integer"
}
},
"required": [
"id",
"price"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
},
"messageId": {
"type": "string"
}
},
"type": "object"
}
},
"runAfter": {},
"type": "ParseJson"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {},
"triggers": {
"manual": {
"inputs": {
"schema": {
"properties": {
"appointment": {
"properties": {
"id": {
"type": "integer"
},
"lines": {
"items": {
"properties": {
"addOn": {
"items": {
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "string"
}
},
"required": [
"id",
"name"
],
"type": "object"
},
"type": "array"
},
"id": {
"type": "integer"
},
"price": {
"type": "integer"
}
},
"required": [
"id",
"price",
"addOn"
],
"type": "object"
},
"type": "array"
}
},
"type": "object"
},
"messageId": {
"type": "string"
}
},
"type": "object"
}
},
"kind": "Http",
"type": "Request"
}
}
},
"parameters": {}
}
Input is :
{
"Size": "2",
"done": "true",
"records": [
{
"Id": "a7g6s0000004GZuAAM",
"NN": "00096411.0",
"Name": "ISOLIN TRADE & INVEST"
},
{
"Id": "a7g6s0000004GZzAAM",
"Number": "00096412.0",
"Name": "ISOLIN"
}
]
}
Spec used:
[
{
"operation": "remove",
"spec": {
"records": {
"*": {
"attributes": " "
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "&",
"records": {
"*": { //iterate on each object of records
"*": { //iterate on each element of object
"$": "Items[#1].Fields[].Name",
"#": "Items[#1].Fields[].Value"
}
}
}
}
}
]
Current Output:
{
"Size": "2",
"done": "true",
"Items": [
{
"Fields": [
{
"Name": "Id"
},
{
"Value": "a7g6s0000004GZuAAM"
},
{
"Name": "NN"
},
{
"Value": "00096411.0"
},
{
"Name": "Name"
},
{
"Value": "ISOLIN TRADE & INVEST"
},
{
"Name": "Id"
},
{
"Value": "a7g6s0000004GZzAAM"
},
{
"Name": "Number"
},
{
"Value": "00096412.0"
},
{
"Name": "Name"
},
{
"Value": "ISOLIN"
}
]
}
]
}
Expected output:
{
"Size": "2",
"done": "true",
"Items": [
{
"Fields": [
{
"Name": "Id",
"Value": "a7g6s0000004GZuAAM"
},
{
"Name": "NN",
"Value": "00096411.0"
},
{
"Name": "Name",
"Value": "ISOLIN TRADE & INVEST"
},
{
"Name": "Id",
"Value": "a7g6s0000004GZzAAM"
},
{
"Name": "Number",
"Value": "00096412.0"
},
{
"Name": "Name",
"Value": "ISOLIN"
}
]
}
]
}
You can seperate those attributes into individual objects through use of Items.&2.[#2]. pattern as prefix such as
[
{
"operation": "shift",
"spec": {
"*": "&",
"records": {
"*": {
"*": {
"$": "Items.&2.[#2].Name",
"#": "Items.&2.[#2].Value"
}
}
}
}
},
{
"operation": "shift",
"spec": {
"*": "&",
"Items": {
"*": {
"*": "Fields[]"
}
}
}
}
]
I am working on to push down the top level field into each element of the array inside it.
I want to push 'sourceEntity' into the each element of 'supportedCountries' array and 'supportedCurrencies' Array
input JSON:
{
"description": "SUPPORTED_COUNTRY_CURRENCY",
"id": "20190902025202944",
"type": "devuae-SUPPORTED_COUNTRY_CURRENCY",
"sourceEntity": "EBI",
"sourceChannel": "COR",
"timestamp": "1567421549887",
"supportedCountries": [
{
"code": "IN",
"name": "India",
"supportedCurrencies": [
{
"code": "JOD",
"name": "JORDANIAN DINAR",
"isLocal": true
}
]
}
]
}
Expected OUTPUT:
{
"description": "SUPPORTED_COUNTRY_CURRENCY",
"id": "20190902025202944",
"type": "devuae-SUPPORTED_COUNTRY_CURRENCY",
"sourceEntity": "EBI",
"sourceChannel": "COR",
"timestamp": "1567421549887",
"supportedCountries": [
{
"EntityID": "EBI",
"code": "IN",
"name": "India",
"supportedCurrencies": [
{
"UnitID": "EBI",
"code": "JOD",
"name": "JORDANIAN DINAR",
"isLocal": true
}
]
}
]
}
Help me to solve this
May be this might help,
[
{
"operation": "shift",
"spec": {
"description": "description",
"id": "id",
"type": "type",
"sourceEntity": "sourceEntity",
"sourceChannel": "sourceChannel",
"timestamp": "timestamp",
"supportedCountries": {
"*": {
//Shifting sourceEntity from the level 1
"#(2,sourceEntity)": "supportedCountries[&1].EntityID",
"code": "supportedCountries[&1].code",
"name": "supportedCountries[&1].name",
"supportedCurrencies": {
"*": {
//Shifting sourceEntity from the level 1
"#(4,sourceEntity)": "supportedCountries[&1].supportedCurrencies[&1].UnitID",
"code": "supportedCountries[&1].supportedCurrencies[&1].code",
"name": "supportedCountries[&1].supportedCurrencies[&1].name",
"isLocal": "supportedCountries[&1].supportedCurrencies[&1].isLocal"
}
}
}
}
}
}
]
I have to index a json to Elastic which look like the below format. My problem is that the key "variable" is array that contains json objects (I thought about "nested" datatype of Elastic) but some of those objects it's possible to contain nested json arrays inside them. (see variable CUSTOMERS).
POST /example_data/data {
"process_name": "TEST_PROCESS",
"process_version ": 0,
"process_id": "1111",
"activity_id": "111",
"name": "update_data",
"username": "testUser",
"datetime": "2018-01-01 10:00:00",
"variables": [{
"name": "ΒΑΝΚ",
"data_type": "STRING",
"value": "EUROBANK"
},{
"name": "CITY",
"data_type": "STRING",
"value": "LONDON"
}, {
"name": "CUSTOMERS",
"data_type": "ENTITY",
"value": [{
"variables": [{
"name": "CUSTOMER_NAME",
"data_type": "STRING",
"value": "JOHN"
}, {
"name": " CUSTOMER_CITY",
"data_type": "STRING",
"value": "LONDON"
}
]
}
]
}, {
"name": "CUSTOMERS",
"data_type": "ENTITY",
"value": [{
"variables": [{
"name": "CUSTOMER_NAME",
"data_type": "STRING",
"value": "ΑΘΗΝΑ"
}, {
"name": " CUSTOMER_CITY ",
"data_type": "STRING",
"value": "LIVERPOOL"
}, {
"name": " CUSTOMER_NUMBER",
"data_type": "STRING",
"value": "1234567890"
}
]
}
]
}
] }
When I'm trying to index it I get the following error
{ "error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Can't merge a non object mapping [variables.value] with an object mapping [variables.value]"
}
],
"type": "illegal_argument_exception",
"reason": "Can't merge a non object mapping [variables.value] with an object mapping [variables.value]" }, "status": 400 }
Mapping
{ "example_data": {
"mappings": {
"data": {
"properties": {
"activity_id": {
"type": "text"
},
"name": {
"type": "text"
},
"process_name": {
"type": "text"
},
"process_version": {
"type": "integer"
}
"process_id": {
"type": "text"
},
"datetime": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"username": {
"type": "text",
"analyzer": "greek"
},
"variables": {
"type": "nested",
"properties": {
"data_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"value": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}}}
When I remove the variable CUSTOMERS that contains the array, then It works properly because there are only json objects.
Is there a way to handle that? Thanks in advance
I have an autocomplete analyser for a field("keywords"). This field is an array of strings. When I query with a search string I want to show first the documents where a single element of the array keywords matches best. The problem is that if a part of the string matches with more elements of the array "keywords", then this document appears before another that has less but better matches. For example, if I have a query with the word "gas station" the returning documents' keywords are these:
"hits": [
{
"_index": "locali_v3",
"_type": "categories",
"_id": "5810767ddc536a03b4761acd",
"_score": 3.1974547,
"_source": {
"keywords": [
"Radio Station",
"Radio Station"
]
}
},
{
"_index": "locali_v3",
"_type": "categories",
"_id": "581076d8dc536a03b4761cc3",
"_score": 3.0407648,
"_source": {
"keywords": [
"Stationery Store",
"Stationery Store"
]
}
},
{
"_index": "locali_v3",
"_type": "categories",
"_id": "5810767ddc536a03b4761ace",
"_score": 2.903595,
"_source": {
"keywords": [
"TV Station",
"TV Station"
]
}
},
{
"_index": "locali_v3",
"_type": "categories",
"_id": "581076cddc536a03b4761c87",
"_score": 2.517158,
"_source": {
"keywords": [
"Praktoreio Ugrwn Kausimwn/Gkaraz",
"Praktoreio Ygrwn Kaysimwn/Gkaraz",
"Praktoreio Ugron Kausimon/Gkaraz",
"Praktoreio Ygron Kaysimon/Gkaraz",
"Πρακτορείο Υγρών Καυσίμων/Γκαράζ",
"Gas Station"
]
}
}
The "Gas Station" is fourth, although it has the best single element matching. Is there a way to tell ElasticSearch that I do not care about how many times "gas" or "station" appears in keywords? I want the max element of the array keywords match as the score factor.
My settings are:
{
"locali": {
"settings": {
"index": {
"creation_date": "1480937810266",
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": "1",
"max_gram": "20"
}
},
"analyzer": {
"keywords": {
"filter": [
"lowercase",
"autocomplete_filter"
],
"char_filter": [
"my_char_filter"
],
"type": "custom",
"tokenizer": "standard"
}
},
"char_filter": {
"my_char_filter": {
"type": "mapping",
"mappings": [
"ί => ι",
"Ί => Ι",
"ή => η",
"Ή => Η",
"ύ => υ",
"Ύ => Υ",
"ά => α",
"Ά => Α",
"έ => ε",
"Έ => Ε",
"ό => ο",
"Ό => Ο",
"ώ => ω",
"Ώ => Ω",
"ϊ => ι",
"ϋ => υ",
"ΐ => ι",
"ΰ => υ"
]
}
}
},
"number_of_shards": "1",
"number_of_replicas": "1",
"uuid": "TJjOt9L9QE2HrsUFHM6zJg",
"version": {
"created": "2040099"
}
}
}
}
}
And the mappings:
{
"locali": {
"mappings": {
"places": {
"properties": {
"formattedCategories": {
"properties": {
"english": {
"type": "string"
},
"greek": {
"type": "string"
}
}
},
"keywords": {
"type": "string",
"analyzer": "keywords"
},
"loc": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
},
"location": {
"properties": {
"formattedAddress": {
"properties": {
"english": {
"type": "string"
},
"greek": {
"type": "string"
}
}
},
"locality": {
"properties": {
"english": {
"type": "string"
},
"greek": {
"type": "string"
}
}
},
"neighbourhood": {
"properties": {
"english": {
"type": "string"
},
"greek": {
"type": "string"
}
}
}
}
},
"name": {
"properties": {
"english": {
"type": "string"
},
"greek": {
"type": "string"
}
}
},
"rating": {
"properties": {
"rating": {
"type": "long"
}
}
},
"seenDetails": {
"type": "long"
},
"verified": {
"type": "long"
}
}
},
"regions": {
"properties": {
"keywords": {
"type": "string",
"analyzer": "keywords"
},
"loc": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
},
"name": {
"properties": {
"english": {
"type": "string"
},
"greek": {
"type": "string"
}
}
},
"type": {
"type": "long"
},
"weight": {
"type": "long"
}
}
},
"categories": {
"properties": {
"keywords": {
"type": "string",
"analyzer": "keywords"
},
"name": {
"properties": {
"english": {
"type": "string"
},
"greek": {
"type": "string"
}
}
},
"weight": {
"type": "long"
}
}
}
}
}
}
Can you post your query here that you are trying here as well.
I tried your example with the following query
{
"query": {"match": {
"keywords": "gas station"
}
}
}
And i got your desired result.
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 3,
"max_score": 0.081366636,
"hits": [
{
"_index": "stack",
"_type": "type",
"_id": "AVjP6QnpdNp-z_ybGd-L",
"_score": 0.081366636,
"_source": {
"keywords": [
"Praktoreio Ugrwn Kausimwn/Gkaraz",
"Praktoreio Ygrwn Kaysimwn/Gkaraz",
"Praktoreio Ugron Kausimon/Gkaraz",
"Praktoreio Ygron Kaysimon/Gkaraz",
"Πρακτορείο Υγρών Καυσίμων/Γκαράζ",
"Gas Station"
]
}
},
{
"_index": "stack",
"_type": "type",
"_id": "AVjP5-u5dNp-z_ybGd-I",
"_score": 0.03182549,
"_source": {
"keywords": [
"Radio Station",
"Radio Station"
]
}
},
{
"_index": "stack",
"_type": "type",
"_id": "AVjP6KiKdNp-z_ybGd-K",
"_score": 0.03182549,
"_source": {
"keywords": [
"TV Station",
"TV Station"
]
}
}
]
}
}
Try this query to see if you are getting desired result. Also you can reply with your mappings, query and ES version if this does't work for you.
Hope this solves your problem. Thanks