Is it possible to send only certain entries of a JSON array?
I have a JSON object defined by the following schema:
"LineGroup": {
"type": "array",
"description": "Line group active",
"items": {
"type": "boolean"
},
"maxItems": 10
}
At the beginning all entries are send. But later on only some entries are changed and only these new values must be updated.
If my syntax at the beginning when I send the full array is:
[{"LineGroup":"False"},{"LineGroup":"True"},...,{"LineGroup":"True"}]
What will be the syntax to only send 1 or 2 entries that have changed in this array? Do I need to resend the whole array?
You could use json patch to update the original json object:
http://jsonpatch.com/
The cool thing is that the patch documents themselves are also json documents.
Related
I'm using the copy data activity in Azure Data Factory to copy data from an API to our data lake for alerting & reporting purposes. The API response is comprised of multiple complex nested JSON arrays with key-value pairs. The API is updated on a quarter-hourly basis and data is only held for 2 days before falling off the stack. The API adopts an oldest-to-newest record structure and so the newest addition to the array would be the final item in the array as opposed to the first.
My requirement is to copy only the most recent record from the API as opposed to the collection - so the 192th reading or item 191 of the array (with the array starting at 0.)
Due to the nature of the solution, there are times when the API isn't being updated as the sensors that collect and send over the data to the server may not be reachable.
The current solution is triggered every 15 minutes and tries a copy data activity of item 191, then 190, then 189 and so on. After 6 attempts it fails and so the record is missed.
current pipeline structure
I have used the mapping tab to specify the items in the array as follows (copy attempt 1 example):
$['meta']['params']['sensors'][*]['name']
$['meta']['sensorReadings'][*]['readings'][191]['dateTime']
$['meta']['sensorReadings'][*]['readings'][191]['value']
Instead of explicitly referencing the array number, I was wondering if it is possible to reference the last item of the array in the above code?
I understand we can use 0 for the first record however I don't understand how to reference the final item. I've tried the following using the 'last' function but am unsure of how to place it:
$['meta']['sensorReadings'][*]['readings'][last]['dateTime']
$['meta']['sensorReadings'][*]['readings']['last']['dateTime']
last['meta']['sensorReadings'][*]['readings']['dateTime']
$['meta']['sensorReadings'][*]['readings']last['dateTime']
Any help or advice on a better way to proceed would be greatly appreciated.
Can you call your API with a Web activity? If so, this pulls the API result into the data pipeline and then apply ADF functions like last to it.
A simple example calling the UK Gov Bank Holidays API:
This returns a resultset that looks like this:
{
"england-and-wales": {
"division": "england-and-wales",
"events": [
{
"title": "New Year’s Day",
"date": "2017-01-02",
"notes": "Substitute day",
"bunting": true
},
{
"title": "Good Friday",
"date": "2017-04-14",
"notes": "",
"bunting": false
},
{
"title": "Easter Monday",
"date": "2017-04-17",
"notes": "",
"bunting": true
},
... etc
You can now apply the last function to is, e.g. using a Set Variable activity:
#string(last(activity('Web1').output['england-and-wales'].events))
Which yields the last bank holiday of 2023:
{
"name": "varWorking",
"value": "{\"title\":\"Boxing Day\",\"date\":\"2023-12-26\",\"notes\":\"\",\"bunting\":true}"
}
Or
#string(last(activity('Web1').output['england-and-wales'].events).date)
I'm using a LogicApp triggered by an HTTP call. The call posts a JSON message which is a single row array. I simply want to extract the single JSON object out of the array so that I can parse it but have spent several hours googling and trying various options to no avail. Here's an example of the array:
[{
"id": "866ef906-5bd8-44d8-af34-0c6906d2dfd7",
"subject": "Engagement-866ef906-5bd8-44d8-af34-0c6906d2dfd7",
"data": {
"$meta": {
"traceparent": "00-dccfde4923181d4196f870385d99cb84-52b8333f100b844c-00"
},
"timestamp": "2021-10-19T17:01:06.334Z",
"correlationId": "866ef906-5bd8-44d8-af34-0c6906d2dfd7",
"fileName": "show.xlsx"
},
"eventType": "File.Uploaded",
"eventTime": "2021-10-19T17:01:07.111Z",
"metadataVersion": "1",
"dataVersion": "1"
}]
Examples of what hasn't worked:
Parse JSON on the array, error: InvalidTemplate when specifiying an array as the schema
For each directly against the http output, error: No dependent actions succeeded.
Any suggestions would be gratefully received.
You have to paste the example that you have provided to 'Use sample payload to generate schema' in the Parse JSON Connector and then you will be able to retrieve each individual object from the sample payload.
You can extract a single JSON object from your array by using its index in square brackets. E.g., in the example below you'd need to use triggerBody()?[0] instead of triggerBody(). 0 is an index of the first element in the array, 1 - of the second, and so on.
Result:
In my Flask server I am receiving a JSON-encoded parameter which is being sent via an HTTP POST from the client application.
Here is an example of what the JSON object looks like. For simplicity, I have kept only the first 2 entries, but the full object contains many more such entries.
[
{
"id": 1,
"start": 7.85,
"end": 9.813,
"text": "Θέλω να",
"words": [
"Θέλω",
"να"
],
"isBeingEditedByUser": false,
"translatedText": "I want to"
},
{
"id": 2,
"start": 9.898,
"end": 13.055,
"text": "Από κάτι το πήραν πολύ άσχημα ο οπαδός του Ολυμπιακού",
"words": [
"Από",
"κάτι",
"το",
"πήραν",
"πολύ",
"άσχημα",
"ο",
"οπαδός",
"του",
"Ολυμπιακού"
],
"isBeingEditedByUser": false,
"translatedText": "Something very bad for Olympiacos fan"
}
]
My understanding is that this JSON structure corresponds to an Array (in Javascript) or a List in Python. In this case, it is an array containing two elements, where each element is itself an object.
However, when I try to use the object on the Flask side, it seems that it has been mapped to a string (rather than a List). Is this normal behavior? I have not been able to find any documentation which states that this is the normal mapping. I would have expected the JSON object to be mapped to a Python List object instead, but this is not happening.
I know that I can use python.loads() myself to convert the string into the appropriate List structure, but I did not expect to have to do this and want to make sure that I am not misunderstanding something here.
Here is a snippet of code which shows the relevant portion in my Flask function:
#app.route('/update_SRT_file', methods=['POST'])
def update_SRT_file():
# Validate the request body contains JSON
if request.is_json:
json_obj = request.get_json()
eprint("update_SRT_file: received JSON object: ")
eprint("Type of received object is", type(json_obj));
else:
eprint("update_SRT_file: Request was not JSON ")
Here is what gets printed out:
23:12:12.771824 update_SRT_file: received JSON object:
23:12:12.771878 Type of received object is **<class 'str'>**
After more investigation, the problem was occuring because the client was JSON encoding the data twice. Upcon removing the additional encoding, it was found that now Flask correctly maps the incoming JSON Array to a python List structure.
Your POST request is sending the JSON as raw text. Flask then receives it as raw text in the request body. Some web application frameworks might automatically parse the text into a JSON-like object or data structure, but Flask does not, at least not out of the box.
Im trying a setup a Microsoft flow. In short, I need to take JSON data retrieved from a device, and parse it so that i could reference it in the Flows below. In order to parse, i need to provide the JSON Schema to Flow. Microsoft Flow has an option to generate it from a sample payload (the results returned from the API call), but it's not generating it correctly. I'm hoping someone can help me. I need the correct JSON Schema.
The data returned from the API:
[
null,
[
{
"user_id": 2003,
"user_label": "Test1"
},
{
"user_id": 2004,
"user_label": "Test2"
}
]
]
Scheme generated in Flow from the above sample payload:
{
"type": "array",
"items": {}
}
I then tried to generate the Schema from just the data. That seemed to work, but when the Flow runs, I get a Json validation error.
Tried generating from just the data like this:
{
"user_id": 2003,
"user_label": "Test1"
}
This generated the scheme like this:
{
"type": "object",
"properties": {
"user_id": {
"type": "number"
},
"user_label": {
"type": "string"
}
}
}
So you have 2 things going on, the nested object array, and the null.
You'll need another Parse JSON after the first Parse JSON. And you'll want to filter out the null before the second Parse JSON.
It took me a while to figure out, but I hope this helps.
Start by adding the Parse JSON step to whatever step is outputting the JSON.
Now, filter the array, make sure you use the 'Expression' when comparing with null.
Add the second Parse JSON, you'll notice that you won't have the option to select the output "Item" of the Filter array step, so select 'Parse JSON' - Item for now (we will change this to use the output of the Filter JSON step in a moment)
The step should automatically change to an 'Apply to each'. In the Parse JSON 2, generate the schema with
[
{
"user_id": 2003,
"user_label": "Test1"
},
{
"user_id": 2004,
"user_label": "Test2"
}
]
Then, modify the 'Select an output from previous steps field' and change it (from the Body of the Parse JSON step) to the Body of the Filter Array step
Finally, add an action after Parse JSON 2 and select one of the fields in Parse JSON 2, this will automatically change that step to a nested Apply to each
You should end up with something like this:
I am trying to define an array element in a JSON Schema. They array contains items from a type that is already defined in the definitions section of the schema.
I have tried:
"properties": {
"userId": {"$ref": "#/definitions/userId"},
"beacons": {
"type": "array",
"items": { "$ref": "#/definitions/beaconSchema" }
}
}
The userId part is parsed with #/definitions/userId. The list items, however, ignore the #/definitions/beaconSchema and allow any old junk in it.
How can I use a JSON schema definition to parse all items in a JSON array?
The schema fragment you posted is correct. I suggest you look for typos in the $ref path and definitions property name. If you don't find the problem there, try posting more of the schema.