Input is as follows:
[
{
"sid": 101,
"sname": "Rahul",
"sem": 1,
"marks": 9
},
{
"sid": 102,
"sname": "Rahul",
"sem": 2,
"marks": 9.5
},
{
"sid": 102,
"sname": "Rahul",
"sem": 3,
"marks": 8
},
{
"sid": 101,
"sname": "Rahul",
"sem": 4,
"marks": 7
}
]
I need to get the total number of marks based on the sid attribute.
You can group by the sid then for each sid sum all the marks.
%dw 2.0
output application/json
---
payload
groupBy ($.sid)
mapObject ($$): sum($.*marks)
Output:
{
"101": 16,
"102": 17.5
}
Note that if you are going to use this information for further processing it will be more efficient to output to application/java instead of JSON.
Alternatively if you prefer to have a list with one element per sid you can use pluck().
%dw 2.0
output application/json
---
payload
groupBy ($.sid)
pluck { sid: ($$), marksTotal: sum($.*marks) }
Output:
[
{
"sid": "101",
"marksTotal": 16
},
{
"sid": "102",
"marksTotal": 17.5
}
]
Related
I need some guidance please.
I have a data structure that looks like this (note that the certResults field is an array, in this case - apple.com and reuters.com, but typically there are more results within the certResults array):
{
"deviceTag": "",
"clientHostName": "555317e186a0",
"dataFormatVersion": 10,
"certResults":
[
{
"hostname": "apple.com",
"port": 443,
"startTime": "2022/07/01 03:50:57.867716",
"endTime": "2022/07/01 03:50:57.960064",
"queryTime": 92.35,
"certificateInfo":
{
"subject":
{
"countryName": "US",
"stateOrProvinceName": "California",
"localityName": "Cupertino",
"organizationName": "Apple Inc.",
"commonName": "apple.com"
},
"certificateIssuer":
{
"countryName": "US",
"organizationName": "Apple Inc.",
"commonName": "Apple Public EV Server ECC CA 1 - G1"
},
"version": 3,
"serialNumber": "6A1D3FA84A43C329F1051060FF4698BA",
"notBefore": "Apr 26 21:58:37 2022 GMT",
"notAfter": "May 26 21:58:36 2023 GMT",
"OCSP":
[
"http://ocsp.apple.com/ocsp03-apevsecc1g101"
],
"crlDistributionPoints":
[
"http://crl.apple.com/apevsecc1g1.crl"
],
"caIssuers":
[
"http://certs.apple.com/apevsecc1g1.der"
],
"subjectAltName":
{
"DNS0": "apple.com"
}
},
"timeLeft": "10 months, 25 days, 18 hours, 7 minutes, 39 seconds",
"percentageUtilization": 16.52
},
{
"hostname": "reuters.com",
"port": 443,
"startTime": "2022/07/01 03:50:57.962692",
"endTime": "2022/07/01 03:50:58.271235",
"queryTime": 308.54,
"certificateInfo":
{
"subject":
{
"commonName": "reuters.com"
},
"certificateIssuer":
{
"countryName": "US",
"organizationName": "Let's Encrypt",
"commonName": "R3"
},
"version": 3,
"serialNumber": "04203F2F15F8194772481DABC1061E213EAB",
"notBefore": "Jun 6 12:54:06 2022 GMT",
"notAfter": "Sep 4 12:54:05 2022 GMT",
"OCSP":
[
"http://r3.o.lencr.org"
],
"caIssuers":
[
"http://r3.i.lencr.org/"
],
"subjectAltName":
{
"DNS0": "reuters.com"
}
},
"timeLeft": "2 months, 3 days, 9 hours, 3 minutes, 7 seconds",
"percentageUtilization": 27.36
}
]
}
This data structure is similar in that it is executed every hour for the same hosts - i.e. each document is the same with updated results in the certResults array.
I'm struggling with the syntax for MongoDB aggregate function.
Using MongoDB's find function, I can collate entries for a specific host by first filtering the dataFormatVersion:10 and then looking at the certResults.hostname field for "reuters.com":
db.certCollection.find({ dataFormatVersion:10, certResults: {$elemMatch: {hostname: "reuters.com"}} }, {_id: 0, certResults: {"hostname.$": 1, "startTime": 1 , "port": 1, "percentageUtilization": 1 }} );
Which presents this information:
{ "certResults" : [ { "hostname" : "reuters.com", "port" : 443, "startTime" : "2022/06/28 16:31:49.919962", "percentageUtilization" : 24.61 } ] }
{ "certResults" : [ { "hostname" : "reuters.com", "port" : 443, "startTime" : "2022/06/28 16:34:55.868512", "percentageUtilization" : 24.61 } ] }
{ "certResults" : [ { "hostname" : "reuters.com", "port" : 443, "startTime" : "2022/06/28 16:57:38.926443", "percentageUtilization" : 24.63 } ] }
{ "certResults" : [ { "hostname" : "reuters.com", "port" : 443, "startTime" : "2022/06/28 17:00:02.359976", "percentageUtilization" : 24.63 } ] }
etc.
I'm not sure how to aggregate data for the percentageUtilization field.
I'd like to find out the average for the percentageUtilization field across the responding documents for this hostname "reuters.com" only. Is there a way to filter on startTime and endTime fields as well?
Similarly, I'd like to find out the average percentageUtilization across all host entries for all responding documents (i.e. reuters.com and apple.com) within a certain timeframe (based on startTime and endTime fields).
Any help would be greatly appreciated!
Thank you!
Aha! The trick I was looking for was to use $unwind for the certResults array.
I am trying to parse out the values I need from this nested JSON data. I need to get from quotas is responders and I need to get from qualified is service_id, codes.
I tried first to get just the quotas but kept getting this error []': no implicit conversion of String into Integer
hash = JSON::parse(response.body)
hash.each do |data|
p data["quotas"]
end
Json data
{
"id": 14706,
"relationships" : [
{
"id": 538
}
]
"quotas": [
{
"id": 48894,
"name": "Test",
"responders": 6,
"qualified": [
{
"service_id": 12,
"codes": [
1,
2,
3,
6,
]
},
{
"service_id": 23,
"pre_codes": [
1,
2
]
}
]
}
]
}
I needed to convert your example into json. Then I could loop the quotas and output the values.
hash = JSON::parse(data.to_json)
hash['quotas'].each do |data|
p data["responders"]
data["qualified"].each do |responder|
p responder['service_id']
p responder['codes']
end
end
Hash in data variable (needed for the sample code to work):
require "json"
data = {
"id": 14706,
"relationships": [
{
"id": 538
}
],
"quotas": [
{
"id": 48894,
"name": "Test",
"responders": 6,
"qualified": [
{
"service_id": 12,
"codes": [
1,
2,
3,
6,
]
},
{
"service_id": 23,
"pre_codes": [
1,
2
]
}
]
}
]
}
How can I get the data out of this array stored in a variant column in Snowflake. I don't care if it's a new table, a view or a query. There is a second column of type varchar(256) that contains a unique ID.
If you can just help me read the "confirmed" data and the "editorIds" data I can probably take it from there. Many thanks!
Output example would be
UniqueID ConfirmationID EditorID
u3kd9 xxxx-436a-a2d7 nupd
u3kd9 xxxx-436a-a2d7 9l34c
R3nDo xxxx-436a-a3e4 5rnj
yP48a xxxx-436a-a477 jTpz8
yP48a xxxx-436a-a477 nupd
[
{
"confirmed": {
"Confirmation": "Entry ID=xxxx-436a-a2d7-3525158332f0: Confirmed order submitted.",
"ConfirmationID": "xxxx-436a-a2d7-3525158332f0",
"ConfirmedOrders": 1,
"Received": "8/29/2019 4:31:11 PM Central Time"
},
"editorIds": [
"xxsJYgWDENLoX",
"JR9bWcGwbaymm3a8v",
"JxncJrdpeFJeWsTbT"
] ,
"id": "xxxxx5AvGgeSHy8Ms6Ytyc-1",
"messages": [],
"orderJson": {
"EntryID": "xxxxx5AvGgeSHy8Ms6Ytyc-1",
"Orders": [
{
"DropShipFlag": 1,
"FromAddressValue": 1,
"OrderAttributes": [
{
"AttributeUID": 548
},
{
"AttributeUID": 553
},
{
"AttributeUID": 2418
}
],
"OrderItems": [
{
"EditorId": "aC3f5HsJYgWDENLoX",
"ItemAssets": [
{
"AssetPath": "https://xxxx573043eac521.png",
"DP2NodeID": "10000",
"ImageHash": "000000000000000FFFFFFFFFFFFFFFFF",
"ImageRotation": 0,
"OffsetX": 50,
"OffsetY": 50,
"PrintedFileName": "aC3f5HsJYgWDENLoX-10000",
"X": 50,
"Y": 52.03909266409266,
"ZoomX": 100,
"ZoomY": 93.75
}
],
"ItemAttributes": [
{
"AttributeUID": 2105
},
{
"AttributeUID": 125
}
],
"ItemBookAttribute": null,
"ProductUID": 52,
"Quantity": 1
}
],
"SendNotificationEmailToAccount": true,
"SequenceNumber": 1,
"ShipToAddress": {
"Addr1": "Addr1",
"Addr2": "0",
"City": "City",
"Country": "US",
"Name": "Name",
"State": "ST",
"Zip": "00000"
}
}
]
},
"orderNumber": null,
"status": "order_placed",
"submitted": {
"Account": "350000",
"ConfirmationID": "xxxxx-436a-a2d7-3525158332f0",
"EntryID": "xxxxx-5AvGgeSHy8Ms6Ytyc-1",
"Key": "D83590AFF0CC0000B54B",
"NumberOfOrders": 1,
"Orders": [
{
"LineItems": [],
"Note": "",
"Products": [
{
"Price": "00.30",
"ProductDescription": "xxxxxint 8x10",
"Quantity": 1
},
{
"Price": "00.40",
"ProductDescription": "xxxxxut Black 8x10",
"Quantity": 1
},
{
"Price": "00.50",
"ProductDescription": "xxxxx"
},
{
"Price": "00.50",
"ProductDescription": "xxxscount",
"Quantity": 1
}
],
"SequenceNumber": "1",
"SubTotal": "00.70",
"Tax": "1.01",
"Total": "00.71"
}
],
"Received": "8/29/2019 4:31:10 PM Central Time"
},
"tracking": null,
"updatedOn": 1.598736670503000e+12
}
]
So, this is how I'd query that exact JSON assuming the data is in column var in table x:
SELECT x.var[0]:confirmed:ConfirmationID::varchar as ConfirmationID,
f.value::varchar as EditorID
FROM x,
LATERAL FLATTEN(input => var[0]:editorIds) f
;
Since your sample output doesn't match the JSON that you provided, I will assume that this is what you need.
Also, as a note, your JSON includes outer [ ] which indicates that the entire JSON string is inside an array. This is the reason for var[0] in my query. If you have multiple records inside that array, then you should remove that. In general, you should exclude those and instead load each record into the table separately. I wasn't sure whether you could make that change, so I just wanted to make note.
I am getting json response from server as below.
[
[{
"ID": 1,
"Date": "11-09-2015",
"Balance": 1496693.00
}, {
"ID": 2,
"Date": "01-10-2015",
"Balance": 1496693.00
}],
[{
"ID": 1,
"Date": "03-09-2000",
"IntAmount": "003.00"
}],
[{
"EmployeeId": "000",
"DesignationName": "deg"
}],
[{
"LoanAmount": "00000.00",
"IntRate": "3.00",
"LoanNo": "56656"
}]
]
I can parse json array with name but in above json there are three arrays without name.
How to parse above json in three different arrays?
If you are positive that the data will always come in the stated format, then you can iterate through the result. See below for example:
main(List<String> args) {
// Define the array of data "object" like this
List<List<Map<String, dynamic>>> arrayOfData = [
[
{"ID": 1, "Date": "11-09-2015", "Balance": 1496693.00},
{"ID": 2, "Date": "01-10-2015", "Balance": 1496693.00}
],
[
{"ID": 1, "Date": "03-09-2000", "IntAmount": "003.00"}
],
[
{"EmployeeId": "000", "DesignationName": "deg"}
],
[
{"LoanAmount": "00000.00", "IntRate": "3.00", "LoanNo": "56656"}
]
];
/*
Iterate through the array of "objects" using forEach,
then, iterate through each resulting array using forEach
*/
arrayOfData.forEach((datasetArray) => datasetArray.forEach((dataset) => print(dataset)));
/*
============== RESULT ========
{ID: 1, Date: 11-09-2015, Balance: 1496693.0}
{ID: 2, Date: 01-10-2015, Balance: 1496693.0}
{ID: 1, Date: 03-09-2000, IntAmount: 003.00}
{EmployeeId: 000, DesignationName: deg}
{LoanAmount: 00000.00, IntRate: 3.00, LoanNo: 56656}
*/
}
I've a table in postgres named: op_user_event_data, which has a column named data, where I store a jsonb, and what I have at the moment is a json like this:
{
"aisles": [],
"taskGroups": [
{
"index": 0,
"tasks": [
{
"index": 1,
"mandatory": false,
"name": "Dados de Linear",
"structuresType": null,
"lines": [
{
"sku": {
"skuId": 1,
"skuName": "Limiano Bola",
"marketId": [
1,
3,
10,
17
],
"productId": 15,
"brandId": [
38,
44
]
},
"taskLineFields": [
{
"tcv": {
"value": "2126474"
},
"columnType": "skuLocalCode",
"columnId": 99
},
{
"tcv": {
"value": null
},
"columnType": "face",
"columnId": 29
},
]
},
{
"sku": {
"skuId": 3,
"skuName": "Limiano Bolinha",
"marketId": [
1,
3,
10,
17
],
"productId": 15,
"brandId": [
38,
44
]
},
"taskLineFields": [
{
"tcv": {
"value": "2545842"
},
"columnType": "skuLocalCode",
"columnId": 99
},
{
"tcv": {
"value": null
},
"columnType": "face",
"columnId": 29
},
]
},
{
"sku": {
"skuId": 5,
"skuName": "Limiano Bola 1/2",
"marketId": [
1,
3,
10,
17
],
"productId": 15,
"brandId": [
38,
44
]
},
"taskLineFields": [
{
"tcv": {
"value": "5127450"
},
"columnType": "skuLocalCode",
"columnId": 99
},
{
"tcv": {
"value": "5.89"
},
"columnType": "rsp",
"columnId": 33
}
]
}
Basically I've an object which has
Aisles [],
taskGroups,
id and name.
Inside the taskGroups as shown in the json, one of the atributes is tasks which is an array, that also have an array called lines which have an array of sku and tasklines.
Basically:
taskGroups -> tasks -> lines -> sku or taskLineFields.
I've tried different queries to get the sku but when I try to get anything further than 'lines' it just came as blank or in some other tries 'cannot call elements from scalar'
Can anyone help me with this issue? Note this is just a sample json.
Anyone knows how make this to work:
I Want all lines where lines->taskLineFields->columnType = 'offer'
All I can do is this, but throwing error on scalar:
SELECT lines->'sku' Produto, lines->'taskLineFields'->'tcv'->>'value' ValorOferta
FROM sd_bel.op_user_event_data,
jsonb_array_elements(data->'taskGroups') taskgroups,
jsonb_array_elements(taskgroups->'tasks') tasks,
jsonb_array_elements(tasks->'columns') columns,
jsonb_array_elements(tasks->'lines') lines
WHERE created_by = 'belteste'
AND lines->'taskLineFields'->>'columnType' = 'offer'
say your data is in some json_column in your table
with t as (
select json_column as xyz from table
),
tg as ( select json_array_elements(xyz->'taskGroups') taskgroups from t),
tsk as (select json_array_elements(taskgroups->'tasks') tasks from tg)
select json_array_elements(tasks->'lines') -> 'sku' as sku from tsk;