Related
Looking to ingest this RESTAPI data to SPLUNK, but having issues with LINE BREAKER, can't seem to discover the correct combination for props.conf.
Also as data is returned in array format without keys, do I need a script to add the keys to the returned array data or can this be achieved using SPLUNK?
N.B.
The keys are returned in the tail of the response.
RESTAPI CALL:
{{base_url}}accounts/{{account}}/{{siteid}}/report?dimensions=queryName,queryType,responseCode,responseCached,coloName,origin,dayOfWeek,tcp,ipVersion,querySizeBucket,responseSizeBucket&metrics=queryCount,uncachedCount,staleCount,responseTimeAvg&limit=2
Any help appreciated.
{
"result": {
"rows": 100,
"data": [
{
"dimensions": [
"college.edu",
"A",
"REFUSED",
"uncached",
"EWR",
"192.0.0.0",
"1",
"0",
"4",
"48-63",
"48-63"
],
"metrics": [
1,
1,
0,
16
]
},
{
"dimensions": [
"school.edu",
"A",
"REFUSED",
"uncached",
"EWR",
"192.0.0.0",
"1",
"0",
"4",
"32-47",
"32-47"
],
"metrics": [
1,
1,
0,
10
]
}
],
"data_lag": 0,
"min": {},
"max": {},
"totals": {
"queryCount": 12,
"responseTimeAvg": 37.28936572607269,
"staleCount": 0,
"uncachedCount": 2147541
},
"query": {
"dimensions": [
"queryName",
"queryType",
"responseCode",
"responseCached",
"coloName",
"origin",
"dayOfWeek",
"tcp",
"ipVersion",
"querySizeBucket",
"responseSizeBucket"
],
"metrics": [
"queryCount",
"uncachedCount",
"staleCount",
"responseTimeAvg"
],
"since": "2022-10-17T04:37:00Z",
"until": "2022-10-17T10:37:00Z",
"limit": 100
}
},
"success": true,
"errors": [],
"messages": []
}
Assuming you want the JSON object to be a single event, the LINE_BREAKER setting should be }([\r\n]+){.
Splunk should have no problems parsing the JSON, but I think there will be problems relating metrics to dimensions because there are multiple sets of data and only one set of keys. Creating a script to combine them seems to be the best option.
We're trying to run an index in Solr (8.9.0 - Schemaless Mode) of a list of items that each contain 1 or 2 arrays of objects, with 1 or more records per array. The sample code below is the json we feed the index:
[
{
"id": 8270861,
"type": "Product",
"title": "Stripped T-shirt"
"tags": [{
"tagId": 218,
"tagIcon": "smile,happy",
"tagHelpText": "",
"tagValue": "grand"
},
{
"tagId": 219,
"tagIcon": "frown,sad",
"tagHelpText": "",
"tagValue": "grand"
}],
"keywords": [
{
"keywordId": 742,
"type": "color"
},
{
"keywordId": 743,
"type": "size"
}]
}
]
2 problems we run into:
PROBLEM 1:
The output of the solr query changes the format of the arrays to this (effectively removing the quotes):
...
"tags": [
"{tagIcon=smile,happy, tagHelpText=, tagId=218, tagValue=grand}",
"{tagIcon=frown,sad, tagHelpText=, tagId=219, tagValue=grand}"
],
"keywords": [
"{type=color, keywordId=742}",
"{type=size, keywordId=743}"
],
...
Is there a way to get the format of the arrays to come back the same way as fed into the index:
"tags": [
{ "tagId": 218, "tagIcon": "smile,happy", "tagHelpText": "", "tagValue": "grand" },
{ "tagId": 219, "tagIcon": "frown,sad", "tagHelpText": "", "tagValue": "grand"}
]
to avoid any conflicts when the value is a comma separated list. Are we missing some definition adjustments in the schema file? If so do we need to define the children of those parent keys(i.e. "tags.tagIcon")?
PROBLEM 2:
The index seems to reject an array with a single element. If we feed it the same json as above, but only one entry in the keywords array (or the tags array):
...
"keywords": [
{
"keywordId": 742,
"type": "color"
}]
...
it throws a code: 400 Unknown operation for the an atomic update: type"
Any suggestions on this would be welcome.
I've got the response from HTTP GET request as JSON object and I want to use that JSON and put other node pass it to the next HTTP request. I got the following response data
{
"settlementFolio": "0002",
"settlementID": 283,
"businessUnitID": 43,
"routeID": 955,
"chargeAmount": 22248.0000,
"paymentAmount": 68560.6000,
"isPaid": false,
"hasValidCrewConfiguration": true,
"settlementDate": "2020-09-21T00:00:00",
"charged": 68560.6000,
"packageRemain": 7500.0000,
"totalBoxes": 280,
"detail": [
{
"settlementTransactionID": 1,
"settlementTransactionDescription": "1 Ventas Netas",
"charge": 0.0,
"payment": 61060.6000,
"isAutomatic": true,
"isForClient": false,
"isCharge": false,
"isChecked": false,
"settlementTransactionClientDetail": [
{
"settlementTransactionID": 3456,
"clientID": 392342,
"saleNote": 11792,
"amount": 30530.3000,
"reference": "",
"clientName": "OXXO NUEVA MAYAPAN",
"isChecked": false
},
{
"settlementTransactionID": 3456,
"clientID": 391322,
"saleNote": 11793,
"amount": 30530.3000,
"reference": "",
"clientName": "OXXO CHICHI SUAREZ",
"isChecked": false
}
]
}
],
"productTypeCounter": [
{
"centralCostID": 1,
"centralCostName": "Refrescos",
"sellUnit": 280,
"cNomina": 0.0000
}
]
}
I need to append a node like this with "settlementTransactionID": 2,
{
"settlementFolio": "0002",
"settlementID": 283,
"businessUnitID": 43,
"routeID": 955,
"chargeAmount": 22248.0000,
"paymentAmount": 68560.6000,
"isPaid": false,
"hasValidCrewConfiguration": true,
"settlementDate": "2020-09-21T00:00:00",
"charged": 68560.6000,
"packageRemain": 7500.0000,
"totalBoxes": 280,
"detail": [
{
"settlementTransactionID": 1,
"settlementTransactionDescription": "1 Ventas Netas",
"charge": 0.0,
"payment": 61060.6000,
"isAutomatic": true,
"isForClient": false,
"isCharge": false,
"isChecked": false,
"settlementTransactionClientDetail": [
{
"settlementTransactionID": 3456,
"clientID": 392342,
"saleNote": 11792,
"amount": 30530.3000,
"reference": "",
"clientName": "OXXO NUEVA MAYAPAN",
"isChecked": false
},
{
"settlementTransactionID": 3456,
"clientID": 391322,
"saleNote": 11793,
"amount": 30530.3000,
"reference": "",
"clientName": "OXXO CHICHI SUAREZ",
"isChecked": false
}
]
},
{
"settlementTransactionID": 2,
"settlementTransactionDescription": "2 Envase Faltante",
"charge": 0.0,
"payment": 7500.0000,
"isAutomatic": true,
"isForClient": false,
"isCharge": false,
"isChecked": false,
"settlementTransactionClientDetail": [
{
"settlementTransactionID": 3461,
"clientID": 0,
"saleNote": null,
"amount": 7500.0000,
"reference": "202|75.00|100.000",
"clientName": "",
"isChecked": false
}
]
}
],
"productTypeCounter": [
{
"centralCostID": 1,
"centralCostName": "Refrescos",
"sellUnit": 280,
"cNomina": 0.0000
}
]
}
I am using something like this
String addMore = '${detail}';
def detail = new groovy.json.JsonSlurper().parseText(addMore);
detail.put(deposito);
vars.put("detail", new groovy.json.JsonBuilder(detail).toPrettyString());
log.info("",vars.get('detail'));
The log shows the following error
020-09-23 09:09:25,857 ERROR o.a.j.e.JSR223PostProcessor: Problem in JSR223 script, JSR223 PostProcessor
javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of method: java.util.ArrayList.put() is applicable for argument types: (org.apache.groovy.json.internal.LazyMap) values: [[settlementTransactionID:11, settlementTransactionDescription:11 Deposito de Efectivo, ...]]
Possible solutions: sum(), pop(), get(int), get(int), sum(java.lang.Object), plus(java.lang.Object)
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:320) ~[groovy-jsr223-3.0.3.jar:3.0.3]
at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:71) ~[groovy-jsr223-3.0.3.jar:3.0.3]
at javax.script.CompiledScript.eval(CompiledScript.java:89) ~[java.scripting:?]
at org.apache.jmeter.util.JSR223TestElement.processFileOrScript(JSR223TestElement.java:222) ~[ApacheJMeter_core.jar:5.3]
at org.apache.jmeter.extractor.JSR223PostProcessor.process(JSR223PostProcessor.java:45) [ApacheJMeter_components.jar:5.3]
at org.apache.jmeter.threads.JMeterThread.runPostProcessors(JMeterThread.java:940) [ApacheJMeter_core.jar:5.3]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:572) [ApacheJMeter_core.jar:5.3]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) [ApacheJMeter_core.jar:5.3]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) [ApacheJMeter_core.jar:5.3]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: groovy.lang.MissingMethodException: No signature of method: java.util.ArrayList.put() is applicable for argument types: (org.apache.groovy.json.internal.LazyMap) values: [[settlementTransactionID:11, settlementTransactionDescription:11 Deposito de Efectivo, ...]]
Possible solutions: sum(), pop(), get(int), get(int), sum(java.lang.Object), plus(java.lang.Object)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:70) ~[groovy-3.0.3.jar:3.0.3]
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46) ~[groovy-3.0.3.jar:3.0.3]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-3.0.3.jar:3.0.3]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) ~[groovy-3.0.3.jar:3.0.3]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139) ~[groovy-3.0.3.jar:3.0.3]
at Script158.run(Script158.groovy:31) ~[?:?]
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317) ~[groovy-jsr223-3.0.3.jar:3.0.3]
... 9 more
We cannot help you as it's unclear what are these ${detail} and ${deposito} mean and contain.
If you get a deposito JMeter Variable which looks like:
{
"settlementTransactionID": 2,
"settlementTransactionDescription": "2 Envase Faltante",
"charge": 0.0,
"payment": 7500.0000,
"isAutomatic": true,
"isForClient": false,
"isCharge": false,
"isChecked": false,
"settlementTransactionClientDetail": [
{
"settlementTransactionID": 3461,
"clientID": 0,
"saleNote": null,
"amount": 7500.0000,
"reference": "202|75.00|100.000",
"clientName": "",
"isChecked": false
}
]
}
and you want to add it to the response of a certain sampler and store the result into another JMeter Variable you can use JSR223 PostProcessor with the following code:
def originalResponse = new groovy.json.JsonSlurper().parse(prev.getResponseData())
originalResponse.detail.add(new groovy.json.JsonSlurper().parseText(vars.get('deposito')))
vars.put('modifiedResponse', new groovy.json.JsonBuilder(originalResponse).toPrettyString())
assuming everything goes well you will get the JSON payload you're looking for in the ${modifiedResponse} JMeter Variable.
More information:
Apache Groovy - Parsing and producing JSON
Apache Groovy - Why and How You Should Use It
How can I get the data out of this array stored in a variant column in Snowflake. I don't care if it's a new table, a view or a query. There is a second column of type varchar(256) that contains a unique ID.
If you can just help me read the "confirmed" data and the "editorIds" data I can probably take it from there. Many thanks!
Output example would be
UniqueID ConfirmationID EditorID
u3kd9 xxxx-436a-a2d7 nupd
u3kd9 xxxx-436a-a2d7 9l34c
R3nDo xxxx-436a-a3e4 5rnj
yP48a xxxx-436a-a477 jTpz8
yP48a xxxx-436a-a477 nupd
[
{
"confirmed": {
"Confirmation": "Entry ID=xxxx-436a-a2d7-3525158332f0: Confirmed order submitted.",
"ConfirmationID": "xxxx-436a-a2d7-3525158332f0",
"ConfirmedOrders": 1,
"Received": "8/29/2019 4:31:11 PM Central Time"
},
"editorIds": [
"xxsJYgWDENLoX",
"JR9bWcGwbaymm3a8v",
"JxncJrdpeFJeWsTbT"
] ,
"id": "xxxxx5AvGgeSHy8Ms6Ytyc-1",
"messages": [],
"orderJson": {
"EntryID": "xxxxx5AvGgeSHy8Ms6Ytyc-1",
"Orders": [
{
"DropShipFlag": 1,
"FromAddressValue": 1,
"OrderAttributes": [
{
"AttributeUID": 548
},
{
"AttributeUID": 553
},
{
"AttributeUID": 2418
}
],
"OrderItems": [
{
"EditorId": "aC3f5HsJYgWDENLoX",
"ItemAssets": [
{
"AssetPath": "https://xxxx573043eac521.png",
"DP2NodeID": "10000",
"ImageHash": "000000000000000FFFFFFFFFFFFFFFFF",
"ImageRotation": 0,
"OffsetX": 50,
"OffsetY": 50,
"PrintedFileName": "aC3f5HsJYgWDENLoX-10000",
"X": 50,
"Y": 52.03909266409266,
"ZoomX": 100,
"ZoomY": 93.75
}
],
"ItemAttributes": [
{
"AttributeUID": 2105
},
{
"AttributeUID": 125
}
],
"ItemBookAttribute": null,
"ProductUID": 52,
"Quantity": 1
}
],
"SendNotificationEmailToAccount": true,
"SequenceNumber": 1,
"ShipToAddress": {
"Addr1": "Addr1",
"Addr2": "0",
"City": "City",
"Country": "US",
"Name": "Name",
"State": "ST",
"Zip": "00000"
}
}
]
},
"orderNumber": null,
"status": "order_placed",
"submitted": {
"Account": "350000",
"ConfirmationID": "xxxxx-436a-a2d7-3525158332f0",
"EntryID": "xxxxx-5AvGgeSHy8Ms6Ytyc-1",
"Key": "D83590AFF0CC0000B54B",
"NumberOfOrders": 1,
"Orders": [
{
"LineItems": [],
"Note": "",
"Products": [
{
"Price": "00.30",
"ProductDescription": "xxxxxint 8x10",
"Quantity": 1
},
{
"Price": "00.40",
"ProductDescription": "xxxxxut Black 8x10",
"Quantity": 1
},
{
"Price": "00.50",
"ProductDescription": "xxxxx"
},
{
"Price": "00.50",
"ProductDescription": "xxxscount",
"Quantity": 1
}
],
"SequenceNumber": "1",
"SubTotal": "00.70",
"Tax": "1.01",
"Total": "00.71"
}
],
"Received": "8/29/2019 4:31:10 PM Central Time"
},
"tracking": null,
"updatedOn": 1.598736670503000e+12
}
]
So, this is how I'd query that exact JSON assuming the data is in column var in table x:
SELECT x.var[0]:confirmed:ConfirmationID::varchar as ConfirmationID,
f.value::varchar as EditorID
FROM x,
LATERAL FLATTEN(input => var[0]:editorIds) f
;
Since your sample output doesn't match the JSON that you provided, I will assume that this is what you need.
Also, as a note, your JSON includes outer [ ] which indicates that the entire JSON string is inside an array. This is the reason for var[0] in my query. If you have multiple records inside that array, then you should remove that. In general, you should exclude those and instead load each record into the table separately. I wasn't sure whether you could make that change, so I just wanted to make note.
I am working on a food delivery app, which uses parse as its backend. I am facing a problem while calling the placeOrder API through
PFCloud.callFunction(inBackground: PlaceOrder, withParameters: params) { (data, err) in}
Please have a look at the JSON which I need to post below.
{
"source": "card_1EVYuOEynlyM6L4SHgBMJYRQ",
"userId": "YjSZYSXEp7",
"data": {
"menuItems": [{
"id": "QSYa2JDcIm",
"title": "Rice With Tibss(Beef)",
"menuTitle": "Rice With Tibss",
"submenuItem": [{
"id": "zaOo6G4KSV",
"name": "Beef",
"price": 12,
"desc": "Fillings?"
}],
"price": 24,
"qty": 1,
"storeId": "yqBCDmzaDP",
"storeName": "Ibex Ethiopian Cusine and Bar",
"orderType": "takeout",
"taxState": 0.0925,
"storeInfo": {
"cart_storeId": "yqBCDmzaDP",
"cart_storeName": "Ibex Ethiopian Cusine and Bar",
"cart_storeImage": "https://res.cloudinary.com/http-get-tolofood-com/image/upload/c_scale,h_199,q_auto,w_270/v1461575640/Ibex_lopx38.jpg",
"cart_storeCuisine": "Ethiopian",
"cart_storeDescription": "We always serve a quality food. We always serve a quality food. We always serve a quality food. We always serve a quality food.",
"cart_storeRating": 3.33,
"cart_storeDelivery": false,
"takeout": true,
"address": "12255 Greenville Ave,Dallas, TX 75243",
"slugname": "TX_DAL_ibex_ethiopian_cuisine_and_bar",
"multiple_location": false,
"cart_storeDeliveryFee": 15,
"cart_storeServes": "Lunch,Dinner",
"busy": false,
"cart_storeSeoSlug": "ibex-ethiopian-cusine-and-bar"
},
"enable": true,
"voice_read_mi_label": "fbgcb",
"voice_read_mi_option": false,
"menuTypeName": "Standard"
}],
"lastOrderType": "takeout",
"searchedAddress": "takeout",
"timeData": {
"day": "06-05-2019",
"time": "12:55 am",
"tz": "America/Los_Angeles"
}
},
"unavailable_option": "restaurant_recommendation"
}
And below is the Swift code which I have used to make pass it.
let storeInfo: Dictionary = [CartStoreId: self.cartStoreId, CartStoreName: self.cartRestaurantName, CartStoreImage: self.cartStoreImage, CartStoreCuisine: self.cartStoreCuisine, CartStoreDescription: self.cartStoreDescription, CartStoreRating: self.cartStoreRating, CartStoreDelivery: self.cartStoreDelivery, Takeout: self.takeOut, Address: self.address, Slugname: self.slugName, MultipleLocation: self.multipleLocation, CartStoreDeliveryFee: self.cartStoreDelivery, CartStoreServes: self.cartStoreServes, Busy: self.busy, CartStoreSeoSlug: self.cartStoreSeoSlug] as Dictionary
let subMenuItem = ["id": "zaOo6G4KSV", "name": "Beef", "price": 12, "desc": "Fillings?", "voice_read_submi_label":"bf", "voice_read_submi_option":false, "disabled": false] as [String: Any]
let ordersDictionary = [
"id" : "1234",
"title" : "Test",
"menuTitle" : "MenuName",
"price" : 23,
"qty" : 2,
"storeId" : 23,
"orderType" : "standard",
"taxState" : 0.22,
"enable" : true,
"menuTypeName" : "Type Name",
"voice_read_mi_label":"fdfs",
"voice_read_mi_option":"false",
"submenuitem": subMenuItem,
"storeInfo": storeInfo
] as Dictionary
let timeData = ["day" : 17-06-2019, "time": "11:00 AM", "tz": "America/Los_Angeles"] as Dictionary
let data = ["menuItems": ordersDictionary, "lastOrderType": "takeout", "searchedAddress": "takeout", "timeData" : timeData] as Dictionary
let params = [UserId: self.userId, "source":"card_1EVYuOEynlyM6L4SHgBMJYRQ", "data": data, "unavailable_option":"restaurant_recommendation","_ApplicationId":"6EuadToYoFGJhI1sX8XnuFBz9tp9l3yH6HxzzXZO", "_JavaScriptKey":"rQkALu9saFtF2oq9yCibyw6mEcs3PVqct3uuP6vg", "_ClientVersion":"js1.6.14", "_InstallationId":"444ec64d-5fcc-7b8e-596e-6be627892c2a",
"_SessionToken":"r:c966376120c8eca77aa63c29d5bebe1a"] as Dictionary
After all this is done I call the parse function like below.
PFCloud.callFunction(inBackground: PlaceOrder, withParameters: params) { (data, err) in
if err != nil {
print(err!)
} else {
print(data!)
}
}
But this gives me error after a few seconds saying
"Error Domain=NSCocoaErrorDomain Code=3840 "JSON text did not start with array or object and option to allow fragments not set." UserInfo={NSDebugDescription=JSON text did not start with array or object and option to allow fragments not set.}"
I have searched the web with the error and made fixes accordingly but still no success. Please help me guys.
I noticed that your params var is not compatible with the JSON you sent, there are more fields and also missing fields. Moreover, menuItems and submenuItem are an Array in your JSON and an Object in your code. It is probably making the cloud code function to fail and you are therefore not receiving back a valid JSON. Try the following and check if it works. In the case it works, just replace the values by your vars.
let params = [
"source": "card_1EVYuOEynlyM6L4SHgBMJYRQ",
"userId": "YjSZYSXEp7",
"data": [
"menuItems": [[
"id": "QSYa2JDcIm",
"title": "Rice With Tibss(Beef)",
"menuTitle": "Rice With Tibss",
"submenuItem": [[
"id": "zaOo6G4KSV",
"name": "Beef",
"price": 12,
"desc": "Fillings?"
]],
"price": 24,
"qty": 1,
"storeId": "yqBCDmzaDP",
"storeName": "Ibex Ethiopian Cusine and Bar",
"orderType": "takeout",
"taxState": 0.0925,
"storeInfo": [
"cart_storeId": "yqBCDmzaDP",
"cart_storeName": "Ibex Ethiopian Cusine and Bar",
"cart_storeImage": "https://res.cloudinary.com/http-get-tolofood-com/image/upload/c_scale,h_199,q_auto,w_270/v1461575640/Ibex_lopx38.jpg",
"cart_storeCuisine": "Ethiopian",
"cart_storeDescription": "We always serve a quality food. We always serve a quality food. We always serve a quality food. We always serve a quality food.",
"cart_storeRating": 3.33,
"cart_storeDelivery": false,
"takeout": true,
"address": "12255 Greenville Ave,Dallas, TX 75243",
"slugname": "TX_DAL_ibex_ethiopian_cuisine_and_bar",
"multiple_location": false,
"cart_storeDeliveryFee": 15,
"cart_storeServes": "Lunch,Dinner",
"busy": false,
"cart_storeSeoSlug": "ibex-ethiopian-cusine-and-bar"
],
"enable": true,
"voice_read_mi_label": "fbgcb",
"voice_read_mi_option": false,
"menuTypeName": "Standard"
]],
"lastOrderType": "takeout",
"searchedAddress": "takeout",
"timeData": [
"day": "06-05-2019",
"time": "12:55 am",
"tz": "America/Los_Angeles"
]
],
"unavailable_option": "restaurant_recommendation"
]