Many thanks for looking at this for me.
I am working in IBM ACE V11 software and in my service, I receive a JSON message.
I need to map this JSON message to a SOAP request via ESQL.
Please see the sample message below:
Incoming JSON message:
"journals": [
{
"journalName": "Plant Species in London",
"journalYear": "2016",
"journalAuthor": [
{
"name": "Julian Bose",
"subject": "botany"
}
{
"name": "Samantha Adams",
"subject": "biology"
},
],
"samplePolling": {
"pollingInterval": 120,
"totalAttempts": 10
}
},
],
"supervisorName": "James Smith"
}
In ESQL I have so far:
For Journal's Name:
SET OutputRoot.SOAP.Body.ns:submitJournal.ns:journalName = InputRoot.JSON.Data.journals.journalName;
For Journal's Year:
SET OutputRoot.SOAP.Body.ns:submitJournal.ns:journalYear = InputRoot.JSON.Data.journals.journalYear;
For Journal's Author, I have a problem. The problem is that there can be 0 to 3 or more authors.
In this case, there are 2 authors.
How do I first check if any authors are present and if so, how many are there and then how to assign each authors' details to SOAP. (All of this in ESQL).
In ESQL I have this so far. But I don't know how to get the "n" value. (n represents no. of authors).
SET OutputRoot.SOAP.Body.ns:submitJournal.ns:journalAuthorValues[n].ns16:AuthorName = InputRoot.JSON.journals.journalAuthor[n].name;
Any and all help is greatly appreciated.
For Journal's Author, I have a problem. The problem is that there can be 0 to 3 or more authors. In this case, there are 2 authors.
You need to iterate over the array of authors, and you are assuming that you need to count the number of authors. But you do not need to. This should work just fine (not tested, may contain syntax errors)
FOR refAuthor AS InputRoot.JSON.Data.journals.(JSON.Array)journalAuthor[] DO
CREATE LASTCHILD OF OutputRoot.SOAP.Body.ns:submitJournal.ns:journalAuthorValues
TYPE NAMEVALUE
IDENTITY ns16:AuthorName
VALUE FIELDVALUE(refAuthor);
END FOR
You should try to avoid using counted loops in ESQL. A FOR statement or a SELECT statement is almost always simpler and better.
Related
I'm using the copy data activity in Azure Data Factory to copy data from an API to our data lake for alerting & reporting purposes. The API response is comprised of multiple complex nested JSON arrays with key-value pairs. The API is updated on a quarter-hourly basis and data is only held for 2 days before falling off the stack. The API adopts an oldest-to-newest record structure and so the newest addition to the array would be the final item in the array as opposed to the first.
My requirement is to copy only the most recent record from the API as opposed to the collection - so the 192th reading or item 191 of the array (with the array starting at 0.)
Due to the nature of the solution, there are times when the API isn't being updated as the sensors that collect and send over the data to the server may not be reachable.
The current solution is triggered every 15 minutes and tries a copy data activity of item 191, then 190, then 189 and so on. After 6 attempts it fails and so the record is missed.
current pipeline structure
I have used the mapping tab to specify the items in the array as follows (copy attempt 1 example):
$['meta']['params']['sensors'][*]['name']
$['meta']['sensorReadings'][*]['readings'][191]['dateTime']
$['meta']['sensorReadings'][*]['readings'][191]['value']
Instead of explicitly referencing the array number, I was wondering if it is possible to reference the last item of the array in the above code?
I understand we can use 0 for the first record however I don't understand how to reference the final item. I've tried the following using the 'last' function but am unsure of how to place it:
$['meta']['sensorReadings'][*]['readings'][last]['dateTime']
$['meta']['sensorReadings'][*]['readings']['last']['dateTime']
last['meta']['sensorReadings'][*]['readings']['dateTime']
$['meta']['sensorReadings'][*]['readings']last['dateTime']
Any help or advice on a better way to proceed would be greatly appreciated.
Can you call your API with a Web activity? If so, this pulls the API result into the data pipeline and then apply ADF functions like last to it.
A simple example calling the UK Gov Bank Holidays API:
This returns a resultset that looks like this:
{
"england-and-wales": {
"division": "england-and-wales",
"events": [
{
"title": "New Year’s Day",
"date": "2017-01-02",
"notes": "Substitute day",
"bunting": true
},
{
"title": "Good Friday",
"date": "2017-04-14",
"notes": "",
"bunting": false
},
{
"title": "Easter Monday",
"date": "2017-04-17",
"notes": "",
"bunting": true
},
... etc
You can now apply the last function to is, e.g. using a Set Variable activity:
#string(last(activity('Web1').output['england-and-wales'].events))
Which yields the last bank holiday of 2023:
{
"name": "varWorking",
"value": "{\"title\":\"Boxing Day\",\"date\":\"2023-12-26\",\"notes\":\"\",\"bunting\":true}"
}
Or
#string(last(activity('Web1').output['england-and-wales'].events).date)
I'm using ConceptNet http://conceptnet.io to try to get related keywords using both their relatedto and edge/query endpoints. The data is awesome, however I've encountered some behaviour I can't figure out.
If you query "relatedto" for the keyword "person" with a limit of 20 on the main site you get:
http://conceptnet.io/c/en/person?rel=/r/RelatedTo&limit=20
Or this list of words:
doll
character
statue
person
servant
body
farmer
child
man
baby
guard
name
doctor
captain
people
neighbour
boy
Pretty awesome right? That's super topical and useful.
On the other hand if you query the API with what appears to be the same query formatted for the API:
http://api.conceptnet.io/related/c/en/person?filter=/c/en&limit=20
Shortened for clarity (see the link above for the full response):
{
"#id": "/c/en/person",
"related": [
{
"#id": "/c/en/person",
"weight": 1.0
},
{
"#id": "/c/en/sean_connery",
"weight": 0.963
},
{
"#id": "/c/en/steve_ballmer",
"weight": 0.962
},
{
"#id": "/c/en/norman_jewison",
"weight": 0.962
},
{
"#id": "/c/en/aretha_franklin",
"weight": 0.962
}
]
}
Huh. What happened there? That's a lot less useful. We got just names and not very related terms.
So my question is: How do I get a similar list?
Are they using some complex edge analysis (using a standard, not relatedto query) to get the relatedterms on the website?
OR
Am I missing something I can't figure out?
Any help much appreciated.
Thanks
To query for existing edges labeled with /r/RelatedTo that contain the node /c/en/person, you should query: http://api.conceptnet.io/query?node=/c/en/person&rel=/r/RelatedTo
The results of that query match the Web site.
The /related endpoint is different, and is only present in the API. It applies some machine learning to predict nodes that are related, whether or not the edge connecting them is already present in ConceptNet. It's better for more specific concepts than "person". Try "teacher" for example: http://api.conceptnet.io/related/c/en/teacher?filter=/c/en&limit=20
I've learned so much using SoapUI, but, I'm just stuck on this one thing. I have the following payload returned:
[
{
"#c": ".CreditPaymentInfo",
"supplementalInfo": null,
"date": "06/30/2015 17:03:50",
"posTxCode": "107535",
"amt": 2.56,
"transactionId": 235087,
"id": 232163,
"cardType": "CREDIT",
"cardHolderName": "SMITH2/JOE",
"expMonthYear": "0119",
"lastFourDigits": "4444",
"approvalCode": "315PNI",
"creditTransactionNumber": "A71A7DB6C2F4"
},
{
"#c": ".CreditPaymentInfo",
"supplementalInfo": null,
"date": "07/01/2015 15:53:29",
"posTxCode": "2097158",
"amt": 58.04,
"transactionId": 235099,
"id": 232176,
"cardType": "CREDIT",
"cardHolderName": "SMITH2/JOE",
"expMonthYear": "0119",
"lastFourDigits": "4444",
"approvalCode": "",
"creditTransactionNumber": null
}
]
I would like to count how many nodes are returned... so, in this case, I would expect that 2 nodes be returned whenever I run this test step in SoapUI.
I was attempting to get this done using the JsonPath Count assertion, but, I just can't see to format it correctly.
Any help would be greatly appreciated!
I have not used JsonPath, but you can do this with XPath ... which works for all older versions too.
Internally SoapUI represents everything as XML. So you could use XPath assertion to check for:
${#ResponseAsXml#count(//*:id)}
and make sure it comes back as
2
Counted successfully with 'JsonPath count' using one of the following (assuming my top level object is an array) :
$
$.
$[*]
If you need to be more specific on the objects you're counting, you can rely only on the 3rd syntax, specifying one of the redundant field. One of the following worked for me :
$[*].fieldName
$[*].['fieldName']
Should return 2 in your case with one of the following :
$[*].#c
$[*].['#c']
$[*].id
$[*].['id']
And so on
This is a JSON format. Just use JsonPath Count. Use $ at the top and 2 at the bottom.
$[index].your.path.here
thus
$[0].date would return "06/30/2015 17:03:50"
and
$[1].date would return "07/01/2015 15:53:29"
I recently found jsonschema and I've been loving using it, however recently I've come across something that I want to do that I just haven't been able to figure out.
What I want to do is to validate that an array must contain an element that matches a schema, but I don't want to have validation fail on other elements that would be in the list.
Say that I have an array like the following:
arr = [
{"some object": True},
False,
{"AnotherObj": "a string this time"},
"test"
]
I want to be able to do something like "validate that arr contains an object that has a property 'some object' that is a boolean, and error if it doesn't, but don't care about other elements."
I don't want it to validate the other items in the list. I just want to make sure that the list contains an element that matches the schema at least once. I also do not know the order which the elements will arrive in the array.
I've tried this already with a schema like:
{"type": "array",
"items": {
"type": "object",
"properties": {
"tool": {
# A schema here to validate tool
},
"required": ["tool"]
}
}
The problem is that it requires every item in the array to have the property "tool", and not what I actually want.
Any help anyone can give me with this would be much appreciated! I've been stumped on this for a really long time with no forward progress.
Thanks!
I've gotten an answer to this question:
The schema used is (where ... B ... is the schema to require):
{
"type": "array",
"not": {
"items": {
"not": {... B ...}
}
}
}
It basically works out to be something like "Ensure that not (items don't match B)". I'm not 100% clear on why this works the way it does, but it does so I figured I'd share it for posterity.
I'm new bee in mongodb.
I made a nested array document like this.
data = {
"title": "mongo community",
"description": "I am a new bee",
"topics": [{
"title": "how to find object in array",
"comments": [{
"description": "desc1"
}]
},
{
"title": "the case to use ensureIndex",
"comments": [{
"description": "before query"
},
{
"description": "If you want"
}
]
}
]
}
after that, put it in the "community"
db.community.insert(data)
so,I would like to accumulate "comments" which topics title is "how to find object in array"
then I tried,
data = db.community.find_one({"title":"mongo community","topics.title":"how to find object in array" } )
the result is
>>> print data
{
u 'topics': [{
u 'comments': [{
u 'description': u 'desc1'
}],
u 'title': u 'how to find object in array'
},
{
u 'comments': [{
u 'description': u 'before query'
},
{
u 'description': u 'If you want'
}],
u 'title': u 'the case to use ensureIndex'
}],
u '_id': ObjectId('4e6ce188d4baa71250000002'),
u 'description': u 'I am a new bee',
u 'title': u 'mongo community'
}
I don't need the topics "the case to use ensureIndex"
Whould you give me any advice.
thx.
It looks like you're embedding topics as an array all in a single document. You should try to avoid returning partial documents frequently from MongoDB. You can do it with the "fields" argument of the find method, but it isn't very easy to work with if you're doing it frequently.
So to solve this you could try to make each topic a separate document. I think that would be easier for you too. If you want to save information about the "community" for forum, put it in a separate collection. For example, you could use the following in the monbodb shell:
// ad a forum:
var forum = {
title:"mongo community",
description:"I am a new bee"
};
db.forums.save(forum);
// add first topic:
var topic = {
title: "how to find object in array",
comments: [ {description:"desc1"} ],
forum:"mongo community"
};
db.topics.save(topic);
// add second topic:
var topic = {
title: "the case to use ensureIndex",
comments: [
{description:"before query"},
{description:"If you want"}
],
forum:"mongo community"
};
db.topics.save(topic);
print("All topics:");
printjson(db.topics.find().toArray());
print("just the 'how to find object in array' topic:")
printjson(db.topics.find({title:"how to find object in array"}).toArray());
Also, see the document Trees In MongoDB about schema design in MongoDB. It happens to be using a similar schema to what you are working with and expands on it for more advanced use cases.
MongoDB operates on documents, that is, the top level documents (the things you save, update, insert, find, and find_one on). Mongo's query language lets you search within embedded objects, but will always return, update, or manipulate one (or more) of these top-level documents.
MongoDB is often called "schema-less," but something more like "(has) flexible schemas" or "(has) per-document schemas" would be a more accurate description. This is a case where your schema design -- having topics embedded directly within a community -- is not working for this particular query. However there are probably other queries that this schema supports more efficiently, like listing the topics within a community in a single query. You might want to consider the queries you want to make and re-design your schema accordingly.
A few notes on MongoDB limitations:
top-level documents are always returned (optionally with only a subset of fields, as #scott noted -- see the mongodb docs on this topic)
each document is limited to 16 megabytes of data (as of version 1.8+), so this schema will not work well if the communities have a long list of topics
For help with schema design, see the mongodb docs on schema design, Kyle Banker's video "Schema Design Basics", and Eliot Horowitz's video "Schema Design at Scale" for an introduction, tips, and considerations.