Choosing right analyzers for Azure Search - azure-cognitive-search

Question

We have created index in Azure Search Service as below:
"analyzers": [
{
"#odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
"name": "SWMLuceneAlongWithCustomHyphenAnalyser",
"tokenizer": "keyword_v2",
"tokenFilters": [
"lowercase"
],
"charFilters": []
}
This analyzer is assigned to a property called "lowerMachineTag". Now when we search using below query, we get expected result:
Query: search=lowerSystemID:/.*it\'s.*/lowerMachineTag:/.*it\'s.*/&$filter=(systemID%20ne%20null)%20and%20(ownerSalesforceRecordID%20eq%20'a0h5B000000gJKfQAM')&$count=true&$top=100&$skip=0
Results:
{
"#odata.context": "https://abcd/indexes('orders-index')/$metadata#docs",
"#odata.count": 4,
"value": [
{
"#search.score": 0.1862714,
"systemID": "*1QXEDL8E2V8MGBY",
"machineTag": "It's me",
"systemIDMachineTag": "*1QXEDL8E2V8MGBY|It's me",
"machineTagSystemID": "It's me|*1QXEDL8E2V8MGBY",
"lowerMachineTag": "it's me",
"lowerSystemID": "*1qxedl8e2v8mgby",
"ownerSalesforceRecordID": "a0h5B000000gJKfQAM",
"parentSalesforceRecordID": "a0h5B000000gJKfQAM"
},
{
"#search.score": 0.16417237,
"systemID": "*1QXEDL8E2V8MGBY",
"machineTag": "It's me",
"systemIDMachineTag": "*1QXEDL8E2V8MGBY|It's me",
"machineTagSystemID": "It's me|*1QXEDL8E2V8MGBY",
"lowerMachineTag": "it's me",
"lowerSystemID": "*1qxedl8e2v8mgby",
"ownerSalesforceRecordID": "a0h5B000000gJKfQAM",
"parentSalesforceRecordID": "a0h5B000000gJKfQAM"
},
{
"#search.score": 0.16417237,
"systemID": "*1QXEDL8E2V8MGBY",
"machineTag": "It's me",
"systemIDMachineTag": "*1QXEDL8E2V8MGBY|It's me",
"machineTagSystemID": "It's me|*1QXEDL8E2V8MGBY",
"lowerMachineTag": "it's me",
"lowerSystemID": "*1qxedl8e2v8mgby",
"ownerSalesforceRecordID": "a0h5B000000gJKfQAM",
"parentSalesforceRecordID": "a0h5B000000gJKfQAM"
},
{
"#search.score": 0.16417237,
"systemID": "*1QXEDL8E2V8MGBY",
"machineTag": "It's me",
"systemIDMachineTag": "*1QXEDL8E2V8MGBY|It's me",
"machineTagSystemID": "It's me|*1QXEDL8E2V8MGBY",
"lowerMachineTag": "it's me",
"lowerSystemID": "*1qxedl8e2v8mgby",
"ownerSalesforceRecordID": "a0h5B000000gJKfQAM",
"parentSalesforceRecordID": "a0h5B000000gJKfQAM"
}
]
}
But what would be the general recommendations for analyzer configuration, if we should have results returned even when we search for lowerMachineTag:/.it./ in additon to above behavior

Answer 1

It seems you are using a regular expression in the search query – for that to work you would have to also add “&queryType=full” to your query string. Otherwise, the entire search term (“lowerSystemID:/.*it\'s.*/lowerMachineTag:/.*it\'s.*/”) would be understood as a simple query, meaning it would be analyzed using the standard analyzer and matched against any searchable field. By adding “&queryType=full” your regex would be understood as such and matched only to the specified fields.
As per your question, if “lowerMachineTag:/.it./” is specified, it wouldn’t match any of the four documents above, as the ‘.’ at the start of the regex would try to match a character before the “it” characters and at least in the four documents above the value of “lowerMachineTag” always starts with “it”.
If you are to remove the starting ‘.’ character, using only “lowerMachineTag:/it./”, it still would not match, because the regex must match the entire token (adding ‘’ would work though: “lowerMachineTag:/it./”).
You can change the analyzer definition to make “/it./” work by also using nGram_v2 token filter, like so:
"analyzers": [
{
"#odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
"name": "SWMLuceneAlongWithCustomHyphenAnalyser",
"tokenizer": "keyword_v2",
"tokenFilters": [
"lowercase", “myNGramTokenFilter”
],
"charFilters": []
},
"tokenFilters":[
{
"name":"myNGramTokenFilter",
"#odata.type":"Microsoft.Azure.Search.NGramTokenFilterV2",
"minGram":1,
"maxGram":100
}
]
This would still make you original query (+ "queryType=full") return the same results and would also return the results when using "lowerMachineTag:/it./".
I hope this helps!