autocomplete with ngrams generates duplicates - solr

I am writing an autocomplete feature in solr. Ideally autocomplete would
display suggestions if target occurs in any of the words, but prefer exact match over KeywordTokenzierFactory ngram edge match, KeywordTokenzierFactory ngram edge match over StandardTokenizer (or UAX29URLEmailTokenizerFactory) ngram edge match
serve the document along with the suggestion.
show unique suggestions only
This is my attempt at autocompleting:
<field name="category" type="string" indexed="true" stored="true" docValues="true"/>
<field name="categoryAutocompleteExactEdge" type="autocomplete_exact_edge" indexed="true" stored="false"/>
<field name="categoryAutocompleteTermsEdge" type="autocomplete_terms_edge" indexed="true" stored="false"/>
<copyField source="category" dest="categoryAutocompleteExactMatch"/>
<copyField source="category" dest="categoryAutocompleteTermsEdge"/>
<fieldType name="autocomplete_exact_edge" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" maxGramSize="30" minGramSize="1"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="autocomplete_terms_edge" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.UAX29URLEmailTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" maxGramSize="30" minGramSize="1"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.UAX29URLEmailTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
</analyzer>
</fieldType>
handler:
<requestHandler name="/suggest_category" class="org.apache.solr.handler.component.SearchHandler">
<lst name="defaults">
<str name="wt">json</str>
<str name="defType">edismax</str>
<str name="rows">5</str>
<str name="fl">category</str>
<str name="qf">category^30 categoryAutocompleteExactEdge^10 categoryAutocompleteTermsEdge</str>
</lst>
</requestHandler>
I think the above handles the order of suggestion in accordance with the first requirement. It also allows you to fetch the document data along with the suggestion by changing fl. The problem I have is the duplication of suggestion.
If there are many documents with category:"GASTROENTEROLOGIST", then it is possible that category: "GASTRO APPOINTMENT" is never served. If faceting is enabled and rows set to 0, then the qf ordering is lost.
I am looking for all in one solution, but it appears to me that serving unique suggestions and also displaying document data is mutually exclusive. For example, if I move the categories to a new core, then the suggestion duplication problem vanishes, because I can force uniqueness. But lookups to the new core can't display additional document info.
This is my first time creating an autocomplete functionality and I am not exactly sure how to tackle it. It would be really helpful if someone experienced could explain the best strategies for handling autocompletion. Is creating a new core for every field with autosuggestion the way to go?

Related

Extracting date from TextField and sorting with it in Solr

I have a a scheme like this in Solr:
<?xml version="1.0" encoding="UTF-8"?>
<schema name="dovecot-fts" version="3.0">
<fieldType name="ytext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="20"/>
<filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<field name="hdr" type="ytext" indexed="true" stored="true"/>
<uniqueKey>id</uniqueKey>
</schema>
And a record looks like this:
{
"id": "339/9821f61c4fa04b62fa030s002df11e39/user#example.com",
"hdr": "...................Some header information...................Date: Fri, 23 Sep 2022 15:24:43 +0300...................Some other header information..................."
}
Now, I want an additional field containing the date in the HDR field to sorting results. To achieve this I tried copying and manipulate hdr field with PatternTokenizerFactory and I can't.
<fieldType name="ts" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory" pattern="Date:\s(\w{3},\s\d{2}\s\w{3,4}\s\d{4}\s\d{2}:\d{2}:\d{2}\s\+\d{4})" group="1"/>
</analyzer>
</fieldType>
<field name="hdr" type="ytext" indexed="true" stored="true"/>
<field name="received" type="ts" indexed="true" stored="false" required="false"/>
<copyField source="hdr" dest="received"/>
So I'm waiting for your help, thanks.
In your schema, you are creating a copy field named received and copying the content of hdr into it. The received field has a specific text analysis but this doesn't change the value of the field. It only changes the way it is indexed. Moreover, you cannot sort documents using a text field.
To achieve your goal you need to transform the value of hdr and copy it into another field of type StrField or Date. It is important to use a non-text type if you want to sort using that field. You can use a custom Update Request Processor Chain: https://solr.apache.org/guide/solr/latest/configuration-guide/update-request-processors.html.
You must insert your custom update chain into the solrconfig.xml. Your best chance is to use the following request processors:
solr.CloneFieldUpdateProcessorFactory
solr.RegexReplaceProcessorFactory
This is an example that you can use as a starting point:
<updateRequestProcessorChain name="extract_hdr_date">
<processor class="solr.CloneFieldUpdateProcessorFactory">
<str name="source">hdr</str>
<str name="dest">hdr_date</str>
</processor>
<processor class="solr.RegexReplaceProcessorFactory">
....
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
Given that you want to use that field for sorting, you might find it useful to enable docvalues on hdr_date: https://solr.apache.org/guide/solr/latest/indexing-guide/docvalues.html

Solr term search not searching all values from multifields value

I have solr field
<field name="AllTitles" type="text_general" indexed="true" stored="false" multiValued="true"/>
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<!-- in this example, we will only use synonyms at query time -->
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Example of Value for AllTitles entered is
AllTitles: [ "Anything", "wuhan coronavirus", "anything" ]
AllTitles: [ "wuhan coronavirus", "anything", "anything" ]
It searches from first index but if any matching term on index other than 1st then it's not searching
For example when I search
q="wuhan coronavirus"
I get 2 results. When I search using field name "AllTitles"
q=AllTitles:"wuhan coronavirus"
I get 7 results correctly.
Can anybody help me identifying the issue?
First, in your SolrConfig.xml check what field has been defined in the "df". In the below example it is "text".
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">text</str>
</requestHandler>
Second, in the schema.xml or managed-schema, whichever you are using, make sure you have copied "AllTitles" to "text". Like this,
<copyField source="AllTitles" dest="text"/>
You might as well test it by adding "AllTitles" to your "df" parameter when you query, before doing all these, like raghu777 has mentioned.

Multi Phrase Query does not find my documents

As part of an upgrade plan from version 6.2.1, I'm setting up a new Solr (7.6.0).
Surprisingly, one of our simple tests failed - inserting a document with some text and then trying to search for it.
The text that was inserted is:
I will think about it.
Request handler configuration:
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">20</int>
<str name="df">text_en</str>
<str name="hl.fragsize">100000</str>
<str name="hl.maxAnalyzedChars">100000</str>
</lst>
</requestHandler>
This is how the field is configured:
<field name="text_en" type="text_en" indexed="true" stored="true" multiValued="true" />
<fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.CommonGramsFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.CommonGramsFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
</analyzer>
</fieldType>
"will" and "it" appears in stopwords_en.txt file.
According to Analysis in the Admin Web App, those are the final tokens that are inserted into the index:
text: i i_will will_think think about about_it
position: 1 1 2 3 4 4
Search for text_en:"I will think about it" doesn't find the document.
Strange thing is, the query "I will think think about it" does work...
Using debugQuery, I noticed a difference compared to our current version.
6.2.1 is using MultiPhraseQuery
7.6.0 is using SpanNearQuery
In version 6.2.1:
"rawquerystring":"text_en:\"I will think about it\"",
"querystring":"text_en:\"I will think about it\"",
"parsedquery":"MultiPhraseQuery(text_en:\"(i i_will) will_think think (about about_it)\")", ...
In 7.6.0 (btw, also in 7.5.0):
"rawquerystring":"text_en:\"I will think about it\"",
"querystring":"text_en:\"I will think about it\"",
"parsedquery":"SpanNearQuery(spanNear([spanOr([spanNear([text_en:i, text_en:will_think], 0, true), spanNear([text_en:i_will, text_en:think], 0, true)]), text_en:about_it], 0, true))", ...
I've found the culprit.
Sharing it to support future googlers.
There was a mistake in the field configuration in schema.xml file.
In the "query" analyzer, instead of "CommonGramsFilterFactory", it should be "CommonGramsQueryFilterFactory".

Front and back EdgeNGrams in Solr

I would like to use EdgeNGramFilterFactory to generate Edge NGrams from front and back. For front I am using
<filter class="solr.EdgeNGramFilterFactory" maxGramSize="20" minGramSize="4"/>
and for back, I am using
<filter class="solr.ReverseStringFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="4" maxGramSize="15"/>
<filter class="solr.ReverseStringFilterFactory"/>
But when they are used together in a single analyzer, the second set of filter factories are acting on the output of the first EdgeNGramFilterFactory.
Is it possible to generate both front and back EdgeNGrams in a single analyzer? Or do I have to create separate analyzers and use copyField to create a field with both the front and back EdgeNGrams?
Update
Example schema as requested in comments below
<fieldType name="text_suggest_edge" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="12"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="12"/>
</analyzer>
</fieldType>
<fieldType name="text_suggest_edge_end" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ReverseStringFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="12"/>
<filter class="solr.ReverseStringFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ReverseStringFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="12"/>
<filter class="solr.ReverseStringFilterFactory"/>
</analyzer>
</fieldType>
<field name="item_name_edge" type="text_suggest_edge" indexed="true" stored="false" multiValued="true"/>
<field name="item_name_edge_end" type="text_suggest_edge_end" indexed="true" stored="false" multiValued="true"/>
<copyField source="item_name" dest="item_name_edge"/>
<copyField source="item_name" dest="item_name_edge_end"/>
Update 2: Including sample input and expected output
Input String
Washington
Required Edge Ngrams
Was, Wash, Washi, ... Washington, ashington, shington, hington ... gton, ton
you could do it in a single analyzer chain if you create your customized version of EdgeNGramFilterFactory (in java, and then plug it into your schema.xml) that creates the additional ngrams from the back.
Otherwise, you are going to need the copyField into an additional field with a separate chain.
I honestly thing the first option is too much trouble, but it is possible for sure.

How to make Solr's spell checker ignore case?

How do you ask the example spellchecker to ignore case ?
I am using all defaults shown in the demo.
Now I see that if I type Ancient, it asks "did you mean ancient ? " What do I do ?
ps : I don't have anything that has the word "spell" in my schema.xml!!!! How is it working ?
The schema should have a field type called "spell" that is used for spell checking. This will lowercase all words used by the spellchecker so you don't have to worry about case. Here is an example of how to use this field type.
Create a field in your schema for spell checking.
<field name="spelling" type="spell" indexed="true" stored="false"/>
And then use a copy field to copy data into this field. The example, the code below will copy the "product_name" field into the spell checker.
<copyField source="product_name" dest="spelling"/>
Edit...
Sorry... I though the "spell" field type was in the default schema. Add this to your schema in the same section as the other fieldType tags.
<fieldType name="spell" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
Please post your solrconfig.xml - I think that will provide a clue.
My best guess will be that solrconfig.xml contains the config for the spellchecker (link) which specifies the field to be used for generating spelling suggestions. That field does not have a LowerCaseFilter in your schema.xml

Resources