I have another question on Azure Search, I have an index called “branchorders-index” (schema attached). I have string field called “lowerCustomerPONbr” with custom analyzer as below:
"analyzers": [
{
"#odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
"name": "SWMLuceneAlongWithCustomHyphenAnalyser",
"tokenizer": "keyword_v2",
"tokenFilters": [
"lowercase"
],
"charFilters": []
}
]
Question: What would be the correct query for searching text “engg - test - umesh”? I want a single document on execution of the query. Here is the query I am trying:
$count=true&$select=lowerCustomerPONbr&search=lowerCustomerPONbr:/.*engg\ \-\ test\ \-\ umesh.*/
{
"#odata.context": "https://abcd/indexes('branchorders-index')/$metadata#docs(lowerCustomerPONbr)",
"#odata.count": 8,
"value": [
{
"#search.score": 0.22780417,
"lowerCustomerPONbr": "engg - test - umesh"
},
{
"#search.score": 0.027440047,
"lowerCustomerPONbr": "sam-clc-test-3"
},
{
"#search.score": 0.025132125,
"lowerCustomerPONbr": "sam-clc-test-4"
},
{
"#search.score": 0.019148104,
"lowerCustomerPONbr": "sam-clc-test-1"
},
{
"#search.score": 0.019148104,
"lowerCustomerPONbr": "030 test 17 april"
},
{
"#search.score": 0.018480092,
"lowerCustomerPONbr": "sam-clc-test-2"
},
{
"#search.score": 0.018480092,
"lowerCustomerPONbr": "dilip-qa-test"
},
{
"#search.score": 0.015009361,
"lowerCustomerPONbr": "030 eng-test"
}
]
}
Santosh, if you add &queryType=full, does this fix the issue?
Related
I have 2 JSON Data 1. Payers 2. Rules. I need to filter Payers JSON data based on PayerId from Rules JSON data.
{
"Payers": [
{
"payerId": "12345",
"name": "Test Payer1"
},
{
"payerId": "23456",
"name": "Test Payer2",
},
{
"payerId": "34567",
"name": "Test Payer3"
}}
Rules JSON file
{
"Rules": [
{
"actions": {
"canCopyRule": true
},
"RuleId": 123,
"description": "Test Rule",
"isDisabled": false,
"Criteria": [
{
"autoSecondaryCriteriaId": 8888,
"criteriaType": { "code": "primaryPayer", "value": "Primary Payer" },
"payerId": ["12345", "34567"]
}
]
}
}]}
I need to filter Payers JSON data based on Rules JSON data if PayerID matches
I need output like below
{
"Payers": [
{
"payerId": "12345",
"name": "Test Payer1"
},
{
"payerId": "34567",
"name": "Test Payer3"
}
}
How to filter?
You can use Array.filter like that (based on your data structure):
const filteredPayers = payersObj.Payers.filter((p) => rulesObj.Rules[0].Criteria[0].payerId.includes(p.payerId));
I can't figure out why your Rules json looks like this, I guess you have multiple rules. If so, you will need to iterate over each rule and invoke includes. Same for Criteria.
Code will check each rule and each critirias
and will return payers if payerId found in any of the given rules of any criteria
const payers = {
"Payers": [
{
"payerId": "12345",
"name": "Test Payer1"
},
{
"payerId": "23456",
"name": "Test Payer2",
},
{
"payerId": "34567",
"name": "Test Payer3"
}]}
const rules = {
"Rules": [
{
"actions": {
"canCopyRule": true
},
"RuleId": 123,
"description": "Test Rule",
"isDisabled": false,
"Criteria": [
{
"autoSecondaryCriteriaId": 8888,
"criteriaType": { "code": "primaryPayer", "value": "Primary Payer" },
"payerId": ["12345", "34567"]
}
]
}
]
}
const data = payers.Payers.filter(payer => rules.Rules.findIndex(rule => rule.Criteria.findIndex(criteria => criteria.payerId.includes(payer.payerId)) != -1) !== -1)
console.log(data)
Consider the following document format which has an array field tasks holding embedded documents
{
"foo": "bar",
"tasks": [
{
"status": "sleep",
"id": "1"
},
{
"status": "active",
"id": "2"
}
]
}
There exists a partial index on key tasks.id
{
"v": 2,
"unique": true,
"key": {
"tasks.id": 1
},
"name": "tasks.id_1",
"partialFilterExpression": {
"tasks.id": {
"$exists": true
}
},
"ns": "zardb.quxcollection"
}
The following $elemMatch query with multiple conditions on the same array element
db.quxcollection.find(
{
"tasks": {
"$elemMatch": {
"id": {
"$eq": "1"
},
"status": {
"$nin": ["active"]
}
}
}
}).explain()
does not seem to use the index
"winningPlan": {
"stage": "COLLSCAN",
"filter": {
"tasks": {
"$elemMatch": {
"$and": [{
"id": {
"$eq": "1"
}
},
{
"status": {
"$not": {
"$eq": "active"
}
}
}
]
}
}
},
"direction": "forward"
}
How can I make the above query use the index? The index does seem to be used via dot notation
db.quxcollection.find({"tasks.id": "1"})
however I need the same array element to match multiple conditions which includes the status field, and the following does not seem to be equivalent to the above $elemMatch based query
db.quxcollection.find({
"tasks.id": "1",
"tasks.status": { "$nin": ["active"] }
})
The way the partial indexes work is it uses the path as a key. With $elemMatch you don't have the path explicitly in the query. If you check it with .explain("allPlansExecution") it is not even considered by the query planner.
To benefit from the index you can specify the path in the query:
db.quxcollection.find(
{
"tasks.id": "1",
"tasks": {
"$elemMatch": {
"id": {
"$eq": "1"
},
"status": {
"$nin": ["active"]
}
}
}
}).explain()
It duplicates part of the elemMatch condition, so the index will be used to get all documents containing tasks of specific id, then it will filter out documents with "active" tasks at fetch stage. I must admit the query doesn't look nice, so may be add some comments to the code with explanations.
I want to pull a number of metrics from Google Analytics API with "Traffic Sources", "Geo Network" and "Audience" dimensions.
So I create the following request. GA Dimensions & Metrics Explorer shows that these metrics & dimensions are compatible. But for some reason, this request returns zero values:
{
"reportRequests": [
{
"viewId": "xxxxxxxx",
"dateRanges": [
{
"startDate": "2020-03-01",
"endDate": "2020-03-11"
}
],
"metrics": [
{
"expression": "ga:sessions"
},
{
"expression": "ga:newUsers"
},
{
"expression": "ga:transactions"
},
{
"expression": "ga:transactionRevenue"
}
],
"dimensions": [
{
"name": "ga:date"
},
{
"name": "ga:campaign"
},
{
"name": "ga:sourceMedium"
},
{
"name": "ga:country"
},
{
"name": "ga:region"
},
{
"name": "ga:city"
},
{
"name": "ga:userAgeBracket"
},
{
"name": "ga:userGender"
},
{
"name": "ga:interestInMarketCategory"
}
]
}
]
}
Although restricted dimensions set shows that data exists:
"dimensions": [
{
"name": "ga:date"
},
{
"name": "ga:campaign"
},
{
"name": "ga:sourceMedium"
},
{
"name": "ga:country"
},
{
"name": "ga:region"
},
{
"name": "ga:city"
}
Why extended dimensions set that shown in 1st example doesn't return data?
Thanks in advance!
Eugene
May be GA doesnot have any information about the user age, gender and in-Market segment (ga:interestInMarketCategory). So when you add these dimensions with others, API returns data for the combination of specified dimensions.
So let's say there are 20 sessions from combination of city = x and region = y. But when you add gender to it, no combination can be made (e.g. city = x and region = y and gender = ?), hence API will return zero response.
Ex.
"docs": [
{
"id": "f37914",
"index_id": "some_index",
"field_1": [
{
"Some value",
"boost": 20.
}
]
},
]
If 'field_1' is matched, then boost by corresponding 'boost' field.
Boost what? the document? the specific field? you can do any of them.
Anyway the way to do it is to user Function Queries:
https://lucene.apache.org/solr/guide/6_6/function-queries.html#FunctionQueries-AvailableFunctions
For example if you want to boost the document (and assuming if the value doesn't match then the score is 0) then you can do something like that:
q:_val_:"if(query($q1), field(boost), 0)"&q1=field_1:"Some Value"
_val_ is just a hook into Solr function query, query returns true if q1 matches, field is a simple function that just return the value of the field it self and if allows us to join the two together.
So what I ended up doing is using lucence payloads and solr 6.6 new DelimitedPayloadTokenFilter feature.
First I created a terms field with the following configuration:
{
"add-field-type": {
"name": "terms",
"stored": "true",
"class": "solr.TextField",
"positionIncrementGap": "100",
"indexAnalyzer": {
"tokenizer": {
"class": "solr.KeywordTokenizerFactory"
},
"filters": [
{
"class": "solr.LowerCaseFilterFactory"
},
{
"class": "solr.DelimitedPayloadTokenFilterFactory",
"encoder": "float",
"delimiter": "|"
}
]
},
"queryAnalyzer": {
"tokenizer": {
"class": "solr.KeywordTokenizerFactory"
},
"filters": [
{
"class": "solr.LowerCaseFilterFactory"
},
{
"class": "solr.SynonymGraphFilterFactory",
"ignoreCase": "true",
"expand": "false",
"tokenizerFactory": "solr.KeywordTokenizerFactory",
"synonyms": "synonyms.txt"
}
]
}
},
"add-field" : {
"name":"terms",
"type":"terms",
"stored": "true",
"multiValued": "true"
}
}
I indexed my documents likes so:
[
{
"id" : "1",
"terms" : [
"some term|10.0",
"another term|60.0"
]
}
,
{
"id" : "2",
"terms" : [
"some term|11.0",
"another term|21.0"
]
}
]
I used solr's functional query support to query for a match on terms and grab the attached boost payload and apply it to the relevancy score:
/solr/payloads/select?indent=on&wt=json&q={!payload_score%20f=ai_terms_wtih_synm_3%20v=$payload_term%20func=max}&fl=id,score&payload_term=some+term
Given my Profile data looks like below, I want to find the profile for combination of userName and productId
and only return the profile with the respective contract for this product.
{
"firstName": "John",
"lastName": "Doe",
"userName": "john.doe#gmail.com",
"language": "NL",
"timeZone": "Europe/Amsterdam",
"contracts": [
{
"contractId": "DEMO1-CONTRACT",
"productId": "ticket-api",
"startDate": ISODate('2016-06-29T09:06:42.391Z'),
"roles": [
{
"name": "Manager",
"permissions": [
{
"activity": "ticket",
"permission": "createTicket"
},
{
"activity": "ticket",
"permission": "updateTicket"
},
{
"activity": "ticket",
"permission": "closeTicket"
}
]
}
]
},
{
"contractId": "DEMO2-CONTRACT",
"productId": "comment-api",
"startDate": ISODate('2016-06-29T10:27:45.899Z'),
"roles": [
{
"name": "Manager",
"permissions": [
{
"activity": "comment",
"permission": "createComment"
},
{
"activity": "comment",
"permission": "updateComment"
},
{
"activity": "comment",
"permission": "deleteComment"
}
]
}
]
}
]
}
I managed to find the solution how to do this from the command line. But I don't seem to find a way how to accomplish this with Morphia (latest version).
db.Profile.aggregate([
{ $match: {"userName": "john.doe#gmail.com"}},
{ $project: {
contracts: {$filter: {
input: '$contracts',
as: 'contract',
cond: {$eq: ['$$contract.productId', "ticket-api"]}
}}
}}
])
This is what I have so far. Any help is most appreciated
Query<Profile> matchQuery = getDatastore().createQuery(Profile.class).field(Profile._userName).equal(userName);
getDatastore()
.createAggregation(Profile.class)
.match(matchQuery)
.project(Projection.expression(??))
Note... meanwhile I found another solution which does not use an aggregation pipeline.
public Optional<Profile> findByUserNameAndContractQuery(String userName, String productId) {
DBObject contractQuery = BasicDBObjectBuilder.start(Contract._productId, productId).get();
Query<Profile> query =
getDatastore()
.createQuery(Profile.class)
.field(Profile._userName).equal(userName)
.filter(Profile._contracts + " elem", contractQuery)
.retrievedFields(true, Profile._contracts + ".$");
return Optional.ofNullable(query.get());
}
I finally found the best way (under assumption I only want to return max. 1 element from array) to filter embedded array.
db.Profile.aggregate([
{ $match: {"userName": "john.doe#gmail.com"}},
{ $unwind: "$contracts"},
{ $match: {"contracts.productId": "comment-api"}}
])
To match according to your first design you could try the projection settings with morphia aggregation pipeline.
Query<Profile> matchQuery = getDatastore().createQuery(Profile.class).field(Profile._userName).equal(userName);
getDatastore()
.createAggregation(Profile.class)
.match(matchQuery)
.project(Projection.expression("$filter", new BasicDBObject()
.append("input", "$contracts")
.append("as", "contract")
.append("cond", new BasicDBObject()
.append("$eq", Arrays.asList('$$contract.productId', "ticket-api")));
Also see the example written by the morphia crew around line 88 at https://github.com/mongodb/morphia/blob/master/morphia/src/test/java/org/mongodb/morphia/aggregation/AggregationTest.java.