Is it possible to get key value pairs from snowflake api instead rowType? - snowflake-cloud-data-platform

I'm working with an API from snowflake and to deal with the json data, I would need to receive data as key-value paired instead of rowType.
I've been searching for results but haven't found any
e.g. A table user with name and email attributes
Name
Email
Kelly
kelly#email.com
Fisher
fisher#email.com
I would request this body:
{
"statement": "SELECT * FROM user",
"timeout": 60,
"database": "DEV",
"schema": "PLACE",
"warehouse": "WH",
"role": "DEV_READER",
"bindings": {
"1": {
"type": "FIXED",
"value": "123"
}
}
}
The results would come like:
{
"resultSetMetaData": {
...
"rowType": [
{ "name": "Name",
...},
{ "name": "Email",
...}
],
},
"data": [
[
"Kelly",
"kelly#email.com"
],
[
"Fisher",
"fisher#email.com"
]
]
}
And the results needed would be:
{
"resultSetMetaData": {
...
"data": [
[
"Name":"Kelly",
"Email":"kelly#email.com"
],
[
"Name":"Fisher",
"Email":"fisher#email.com"
]
]
}
Thank you for any inputs

The output is not valid JSON, but the return can arrive in a slightly different format:
{
"resultSetMetaData": {
...
"data":
[
{
"Name": "Kelly",
"Email": "kelly#email.com"
},
{
"Name": "Fisher",
"Email": "fisher#email.com"
}
]
}
}
To get the API to send it that way, you can change the SQL from select * to:
select object_construct(*) as KVP from "USER";
You can also specify the names of the keys using:
select object_construct('NAME', "NAME", 'EMAIL', EMAIL) from "USER";
The object_construct function takes an arbitrary number of parameters, as long as they're even, so:
object_construct('KEY1', VALUE1, 'KEY2', VALUE2, <'KEY_N'>, <VALUE_N>)

Related

How to find a max value of a specific key in a dictionary in a list in a document in mongodb?

I want to use an aggregation to get the highest value of a specified key that's in a dict field that's in a list field that's in a document that's in a mongodb collection.
Here's some example data
[
{
"name": "hi",
"hist": [
{
"username": "bill",
},
{
"username": "jack",
"changed_from": 127
}
]
},
{
"name": "member1",
"hist": [
{
"username": "asdf",
"changed_from": 123
},
{
"username": "duhby",
"changed_from": 126
}
]
},
{
"name": "member5",
"hist": [
{
"username": "duhby",
"changed_from": 150
},
{
"username": "test",
"changed_from": 123
},
{
"username": "duhby",
"changed_from": 125
}
]
}
]
I want to be able to put in duhby as the username, for example, and get at least a list of results I can then easily get the maximum value of, with the maximum value in this case being 150.
I tried using an aggregate group but got stuck when trying to only get the data from the specific username, and not just all documents that had that username in the hist field.
db.collection.aggregate([
{
$group: {
"_id": "$hist.duhby",
update_time: {
$max: "$hist.changed_from"
}
}
}
])
With the example data shown earlier, this returns:
[
{
"_id": [],
"update_time": [
150,
123,
125
]
}
]
However, this isn't useful because it shows every changed_from value when I want it to only show (and sort by) the ones with the username specified. Expected result:
[
{
"_id": [],
"update_time": [
150,
125
]
}
]
I also want to be able to get the original document and maybe have the name field as the id in the aggregation, but the id is currently returned as an empty list.
I realized here that what I needed is just a find, not an aggregation, because I was just trying to see if any document existed that had the embedded document with a key value pair of a certain value. The following does what I was looking for:
db.collection.find({
"hist": {
$elemMatch: {
username: "duhby",
changed_from: {
"$lte": 123
}
}
}
})

How to filter JSON data based on another JSON data in typescript

I have 2 JSON Data 1. Payers 2. Rules. I need to filter Payers JSON data based on PayerId from Rules JSON data.
{
"Payers": [
{
"payerId": "12345",
"name": "Test Payer1"
},
{
"payerId": "23456",
"name": "Test Payer2",
},
{
"payerId": "34567",
"name": "Test Payer3"
}}
Rules JSON file
{
"Rules": [
{
"actions": {
"canCopyRule": true
},
"RuleId": 123,
"description": "Test Rule",
"isDisabled": false,
"Criteria": [
{
"autoSecondaryCriteriaId": 8888,
"criteriaType": { "code": "primaryPayer", "value": "Primary Payer" },
"payerId": ["12345", "34567"]
}
]
}
}]}
I need to filter Payers JSON data based on Rules JSON data if PayerID matches
I need output like below
{
"Payers": [
{
"payerId": "12345",
"name": "Test Payer1"
},
{
"payerId": "34567",
"name": "Test Payer3"
}
}
How to filter?
You can use Array.filter like that (based on your data structure):
const filteredPayers = payersObj.Payers.filter((p) => rulesObj.Rules[0].Criteria[0].payerId.includes(p.payerId));
I can't figure out why your Rules json looks like this, I guess you have multiple rules. If so, you will need to iterate over each rule and invoke includes. Same for Criteria.
Code will check each rule and each critirias
and will return payers if payerId found in any of the given rules of any criteria
const payers = {
"Payers": [
{
"payerId": "12345",
"name": "Test Payer1"
},
{
"payerId": "23456",
"name": "Test Payer2",
},
{
"payerId": "34567",
"name": "Test Payer3"
}]}
const rules = {
"Rules": [
{
"actions": {
"canCopyRule": true
},
"RuleId": 123,
"description": "Test Rule",
"isDisabled": false,
"Criteria": [
{
"autoSecondaryCriteriaId": 8888,
"criteriaType": { "code": "primaryPayer", "value": "Primary Payer" },
"payerId": ["12345", "34567"]
}
]
}
]
}
const data = payers.Payers.filter(payer => rules.Rules.findIndex(rule => rule.Criteria.findIndex(criteria => criteria.payerId.includes(payer.payerId)) != -1) !== -1)
console.log(data)

useQuery's onCompleted being called with cached value

Hopefully I can articulate this question clearly without too much code as it's difficult to extract the pieces from my codebase.
I was observing odd behavior yesterday with useQuery that I can't seem to understand. I think I understand Apollo's cache pretty well but this particular behavior doesn't make sense to me. I have a query that looks something like this:
query {
reservations {
priceBreakdown {
sections {
id
name
total
}
}
}
}
The schema is something like:
type Query {
reservations: [Reservation]
}
type Reservation {
priceBreakdown: PriceBreakdown
}
type PriceBreakdown {
sections: [Section]
}
type Section {
id: String
name: String
total: Float
}
That id on Section is not a proper ID and, in fact, is not unique. It's just a string and all PriceBreakdowns have a list of Sections that contain the same ID. I've pointed this out to the backend folks and it's being fixed but I realize this causes incorrect caching with Apollo since there will be collisions w.r.t. __typename and id. My confusion comes from how onCompleted is called. I noticed when doing
const { data } = useQuery(myQuery, {
onCompleted: console.log
})
that when the network call returns, all PriceBreakdowns are unique and correct, as they should be. But when onCompleted is called with what I thought would be that same API data, it's different and seems to reflect the cached values. In case that's confusing, here are the two results. First is straight from the API and second is the log from onCompleted:
// api results
"data": [
{
"id": "92267",
"price_breakdown": {
"sections": [
{
"name": "Reservation",
"total": "$60.00",
"id": "RESERVATION"
},
{
"name": "Promotions and Fees",
"total": null,
"id": "PROMOTIONS_AND_FEES"
},
{
"name": "Total",
"total": "$51.00",
"id": "HOST_TOTAL"
}
]
}
},
{
"id": "92266",
"price_breakdown": {
"sections": [
{
"name": "Reservation",
"total": "$30.00",
"id": "RESERVATION"
},
{
"name": "Promotions and Fees",
"total": null,
"id": "PROMOTIONS_AND_FEES"
},
{
"name": "Total",
"total": "$25.50",
"id": "HOST_TOTAL"
}
]
}
}
]
// onCompleted log
"data": [
{
"id": "92267",
"price_breakdown": {
"sections": [
{
"name": "Reservation",
"total": "$60.00",
"id": "RESERVATION"
},
{
"name": "Promotions and Fees",
"total": null,
"id": "PROMOTIONS_AND_FEES"
},
{
"name": "Total",
"total": "$51.00",
"id": "HOST_TOTAL"
}
]
}
},
{
"id": "92266",
"price_breakdown": {
"sections": [
{
"name": "Reservation",
"total": "$60.00",
"id": "RESERVATION"
},
{
"name": "Promotions and Fees",
"total": null,
"id": "PROMOTIONS_AND_FEES"
},
{
"name": "Total",
"total": "$51.00",
"id": "HOST_TOTAL"
}
]
}
}
]
As you can see, in the onCompleted log, the Sections that had the same ID as Sections from the previous record are duplicated, suggesting Apollo is rebuilding the payload from cache and calling onCompleted with that. Is that what's happening? If I set the fetchPolicy to no-cache, the results are correct, but of course that's just a patch for the problem. I want to better understand Apollo because I thought I understood and now I see something unintuitive. I wouldn't have expected onCompleted to be called with something built from the cache. Thanks in advance.

Extended dimensions set doesn't return data

I want to pull a number of metrics from Google Analytics API with "Traffic Sources", "Geo Network" and "Audience" dimensions.
So I create the following request. GA Dimensions & Metrics Explorer shows that these metrics & dimensions are compatible. But for some reason, this request returns zero values:
{
"reportRequests": [
{
"viewId": "xxxxxxxx",
"dateRanges": [
{
"startDate": "2020-03-01",
"endDate": "2020-03-11"
}
],
"metrics": [
{
"expression": "ga:sessions"
},
{
"expression": "ga:newUsers"
},
{
"expression": "ga:transactions"
},
{
"expression": "ga:transactionRevenue"
}
],
"dimensions": [
{
"name": "ga:date"
},
{
"name": "ga:campaign"
},
{
"name": "ga:sourceMedium"
},
{
"name": "ga:country"
},
{
"name": "ga:region"
},
{
"name": "ga:city"
},
{
"name": "ga:userAgeBracket"
},
{
"name": "ga:userGender"
},
{
"name": "ga:interestInMarketCategory"
}
]
}
]
}
Although restricted dimensions set shows that data exists:
"dimensions": [
{
"name": "ga:date"
},
{
"name": "ga:campaign"
},
{
"name": "ga:sourceMedium"
},
{
"name": "ga:country"
},
{
"name": "ga:region"
},
{
"name": "ga:city"
}
Why extended dimensions set that shown in 1st example doesn't return data?
Thanks in advance!
Eugene
May be GA doesnot have any information about the user age, gender and in-Market segment (ga:interestInMarketCategory). So when you add these dimensions with others, API returns data for the combination of specified dimensions.
So let's say there are 20 sessions from combination of city = x and region = y. But when you add gender to it, no combination can be made (e.g. city = x and region = y and gender = ?), hence API will return zero response.

How to filter embedded array in mongo document with morphia

Given my Profile data looks like below, I want to find the profile for combination of userName and productId
and only return the profile with the respective contract for this product.
{
"firstName": "John",
"lastName": "Doe",
"userName": "john.doe#gmail.com",
"language": "NL",
"timeZone": "Europe/Amsterdam",
"contracts": [
{
"contractId": "DEMO1-CONTRACT",
"productId": "ticket-api",
"startDate": ISODate('2016-06-29T09:06:42.391Z'),
"roles": [
{
"name": "Manager",
"permissions": [
{
"activity": "ticket",
"permission": "createTicket"
},
{
"activity": "ticket",
"permission": "updateTicket"
},
{
"activity": "ticket",
"permission": "closeTicket"
}
]
}
]
},
{
"contractId": "DEMO2-CONTRACT",
"productId": "comment-api",
"startDate": ISODate('2016-06-29T10:27:45.899Z'),
"roles": [
{
"name": "Manager",
"permissions": [
{
"activity": "comment",
"permission": "createComment"
},
{
"activity": "comment",
"permission": "updateComment"
},
{
"activity": "comment",
"permission": "deleteComment"
}
]
}
]
}
]
}
I managed to find the solution how to do this from the command line. But I don't seem to find a way how to accomplish this with Morphia (latest version).
db.Profile.aggregate([
{ $match: {"userName": "john.doe#gmail.com"}},
{ $project: {
contracts: {$filter: {
input: '$contracts',
as: 'contract',
cond: {$eq: ['$$contract.productId', "ticket-api"]}
}}
}}
])
This is what I have so far. Any help is most appreciated
Query<Profile> matchQuery = getDatastore().createQuery(Profile.class).field(Profile._userName).equal(userName);
getDatastore()
.createAggregation(Profile.class)
.match(matchQuery)
.project(Projection.expression(??))
Note... meanwhile I found another solution which does not use an aggregation pipeline.
public Optional<Profile> findByUserNameAndContractQuery(String userName, String productId) {
DBObject contractQuery = BasicDBObjectBuilder.start(Contract._productId, productId).get();
Query<Profile> query =
getDatastore()
.createQuery(Profile.class)
.field(Profile._userName).equal(userName)
.filter(Profile._contracts + " elem", contractQuery)
.retrievedFields(true, Profile._contracts + ".$");
return Optional.ofNullable(query.get());
}
I finally found the best way (under assumption I only want to return max. 1 element from array) to filter embedded array.
db.Profile.aggregate([
{ $match: {"userName": "john.doe#gmail.com"}},
{ $unwind: "$contracts"},
{ $match: {"contracts.productId": "comment-api"}}
])
To match according to your first design you could try the projection settings with morphia aggregation pipeline.
Query<Profile> matchQuery = getDatastore().createQuery(Profile.class).field(Profile._userName).equal(userName);
getDatastore()
.createAggregation(Profile.class)
.match(matchQuery)
.project(Projection.expression("$filter", new BasicDBObject()
.append("input", "$contracts")
.append("as", "contract")
.append("cond", new BasicDBObject()
.append("$eq", Arrays.asList('$$contract.productId', "ticket-api")));
Also see the example written by the morphia crew around line 88 at https://github.com/mongodb/morphia/blob/master/morphia/src/test/java/org/mongodb/morphia/aggregation/AggregationTest.java.

Resources