Solr term search not searching all values from multifields value - solr

I have solr field
<field name="AllTitles" type="text_general" indexed="true" stored="false" multiValued="true"/>
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<!-- in this example, we will only use synonyms at query time -->
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Example of Value for AllTitles entered is
AllTitles: [ "Anything", "wuhan coronavirus", "anything" ]
AllTitles: [ "wuhan coronavirus", "anything", "anything" ]
It searches from first index but if any matching term on index other than 1st then it's not searching
For example when I search
q="wuhan coronavirus"
I get 2 results. When I search using field name "AllTitles"
q=AllTitles:"wuhan coronavirus"
I get 7 results correctly.
Can anybody help me identifying the issue?

First, in your SolrConfig.xml check what field has been defined in the "df". In the below example it is "text".
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">text</str>
</requestHandler>
Second, in the schema.xml or managed-schema, whichever you are using, make sure you have copied "AllTitles" to "text". Like this,
<copyField source="AllTitles" dest="text"/>
You might as well test it by adding "AllTitles" to your "df" parameter when you query, before doing all these, like raghu777 has mentioned.

Related

autocomplete with ngrams generates duplicates

I am writing an autocomplete feature in solr. Ideally autocomplete would
display suggestions if target occurs in any of the words, but prefer exact match over KeywordTokenzierFactory ngram edge match, KeywordTokenzierFactory ngram edge match over StandardTokenizer (or UAX29URLEmailTokenizerFactory) ngram edge match
serve the document along with the suggestion.
show unique suggestions only
This is my attempt at autocompleting:
<field name="category" type="string" indexed="true" stored="true" docValues="true"/>
<field name="categoryAutocompleteExactEdge" type="autocomplete_exact_edge" indexed="true" stored="false"/>
<field name="categoryAutocompleteTermsEdge" type="autocomplete_terms_edge" indexed="true" stored="false"/>
<copyField source="category" dest="categoryAutocompleteExactMatch"/>
<copyField source="category" dest="categoryAutocompleteTermsEdge"/>
<fieldType name="autocomplete_exact_edge" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" maxGramSize="30" minGramSize="1"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="autocomplete_terms_edge" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.UAX29URLEmailTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" maxGramSize="30" minGramSize="1"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.UAX29URLEmailTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
</analyzer>
</fieldType>
handler:
<requestHandler name="/suggest_category" class="org.apache.solr.handler.component.SearchHandler">
<lst name="defaults">
<str name="wt">json</str>
<str name="defType">edismax</str>
<str name="rows">5</str>
<str name="fl">category</str>
<str name="qf">category^30 categoryAutocompleteExactEdge^10 categoryAutocompleteTermsEdge</str>
</lst>
</requestHandler>
I think the above handles the order of suggestion in accordance with the first requirement. It also allows you to fetch the document data along with the suggestion by changing fl. The problem I have is the duplication of suggestion.
If there are many documents with category:"GASTROENTEROLOGIST", then it is possible that category: "GASTRO APPOINTMENT" is never served. If faceting is enabled and rows set to 0, then the qf ordering is lost.
I am looking for all in one solution, but it appears to me that serving unique suggestions and also displaying document data is mutually exclusive. For example, if I move the categories to a new core, then the suggestion duplication problem vanishes, because I can force uniqueness. But lookups to the new core can't display additional document info.
This is my first time creating an autocomplete functionality and I am not exactly sure how to tackle it. It would be really helpful if someone experienced could explain the best strategies for handling autocompletion. Is creating a new core for every field with autosuggestion the way to go?

Extracting date from TextField and sorting with it in Solr

I have a a scheme like this in Solr:
<?xml version="1.0" encoding="UTF-8"?>
<schema name="dovecot-fts" version="3.0">
<fieldType name="ytext" class="solr.TextField" autoGeneratePhraseQueries="true" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="3" maxGramSize="20"/>
<filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
<filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="1" catenateWords="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<field name="hdr" type="ytext" indexed="true" stored="true"/>
<uniqueKey>id</uniqueKey>
</schema>
And a record looks like this:
{
"id": "339/9821f61c4fa04b62fa030s002df11e39/user#example.com",
"hdr": "...................Some header information...................Date: Fri, 23 Sep 2022 15:24:43 +0300...................Some other header information..................."
}
Now, I want an additional field containing the date in the HDR field to sorting results. To achieve this I tried copying and manipulate hdr field with PatternTokenizerFactory and I can't.
<fieldType name="ts" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory" pattern="Date:\s(\w{3},\s\d{2}\s\w{3,4}\s\d{4}\s\d{2}:\d{2}:\d{2}\s\+\d{4})" group="1"/>
</analyzer>
</fieldType>
<field name="hdr" type="ytext" indexed="true" stored="true"/>
<field name="received" type="ts" indexed="true" stored="false" required="false"/>
<copyField source="hdr" dest="received"/>
So I'm waiting for your help, thanks.
In your schema, you are creating a copy field named received and copying the content of hdr into it. The received field has a specific text analysis but this doesn't change the value of the field. It only changes the way it is indexed. Moreover, you cannot sort documents using a text field.
To achieve your goal you need to transform the value of hdr and copy it into another field of type StrField or Date. It is important to use a non-text type if you want to sort using that field. You can use a custom Update Request Processor Chain: https://solr.apache.org/guide/solr/latest/configuration-guide/update-request-processors.html.
You must insert your custom update chain into the solrconfig.xml. Your best chance is to use the following request processors:
solr.CloneFieldUpdateProcessorFactory
solr.RegexReplaceProcessorFactory
This is an example that you can use as a starting point:
<updateRequestProcessorChain name="extract_hdr_date">
<processor class="solr.CloneFieldUpdateProcessorFactory">
<str name="source">hdr</str>
<str name="dest">hdr_date</str>
</processor>
<processor class="solr.RegexReplaceProcessorFactory">
....
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
Given that you want to use that field for sorting, you might find it useful to enable docvalues on hdr_date: https://solr.apache.org/guide/solr/latest/indexing-guide/docvalues.html

Why solr keywordtokenizerfactory field query response is taking so long

I have made the type definition in Solr \conf\managed-schema like below. My core is very huge i.e. ~30 million documents, ~20 GB of data/indexed size, ~12 fields, and JVM memory allocation is -Xms10g, -Xmx10g
<field name="Address" type="text_keyword" default="" multiValued="false" indexed="true" stored="true"/>
<fieldType name="text_keyword" class="solr.TextField" positionIncrementGap="100" multiValued="true">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
And a /update request handler at conf\solrconfig.xml,
<requestHandler name="/update" class="solr.UpdateRequestHandler" >
<lst name="defaults">
<str name="update.chain">dedupe</str>
</lst>
in conf\solrconfig.xml
<updateRequestProcessorChain name="dedupe">
<processor class="solr.processor.SignatureUpdateProcessorFactory">
<bool name="enabled">true</bool>
<str name="signatureField">id</str>
<str name="fields">Address,Field1,Field2,Field3,Field4,Field5,Field6,Field7,Field8,Field9,Field10,Field10</str>
<str name="signatureClass">solr.processor.Lookup3Signature</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
Here is my search field query example:
http://localhost:8983/solr/mycore/select?indent=true&q.op=OR&q=*:*&fq=((Address:(*premier*)) OR (Field1:(*premier*)) OR (Field2:(*premier*)) OR (Field3:(*premier*)) OR (Field4:(*premier*)) OR (Field5:(*premier*)) OR (Field6:(*premier*)) OR (Field7:(*premier*)) OR (Field8:(*premier*)) OR (Field9:(*premier*)) OR (Field10:(*premier*)) OR (Field11:(*premier*)))
AND ((Address:(*1\:34*)) OR (Field1:(*1\:34*)) OR (Field2:(*1\:34*)) OR (Field3:(*1\:34*)) OR (Field4:(*1\:34*)) OR (Field5:(*1\:34*)) OR (Field6:(*1\:34*)) OR (Field7:(*1\:34*)) OR (Field8:(*1\:34*)) OR (Field9:(*1\:34*)) OR (Field10:(*1\:34*)) OR (Field11:(*1\:34*)))
&rows=10&start=0&wt=json
The special characters are escaped with \ and I am trying filter the records like LIKE clause in RDBMS. The above query is working fine but the response is taking so long. What can I do to speed it up?

Multi Phrase Query does not find my documents

As part of an upgrade plan from version 6.2.1, I'm setting up a new Solr (7.6.0).
Surprisingly, one of our simple tests failed - inserting a document with some text and then trying to search for it.
The text that was inserted is:
I will think about it.
Request handler configuration:
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">20</int>
<str name="df">text_en</str>
<str name="hl.fragsize">100000</str>
<str name="hl.maxAnalyzedChars">100000</str>
</lst>
</requestHandler>
This is how the field is configured:
<field name="text_en" type="text_en" indexed="true" stored="true" multiValued="true" />
<fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.CommonGramsFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.CommonGramsFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
</analyzer>
</fieldType>
"will" and "it" appears in stopwords_en.txt file.
According to Analysis in the Admin Web App, those are the final tokens that are inserted into the index:
text: i i_will will_think think about about_it
position: 1 1 2 3 4 4
Search for text_en:"I will think about it" doesn't find the document.
Strange thing is, the query "I will think think about it" does work...
Using debugQuery, I noticed a difference compared to our current version.
6.2.1 is using MultiPhraseQuery
7.6.0 is using SpanNearQuery
In version 6.2.1:
"rawquerystring":"text_en:\"I will think about it\"",
"querystring":"text_en:\"I will think about it\"",
"parsedquery":"MultiPhraseQuery(text_en:\"(i i_will) will_think think (about about_it)\")", ...
In 7.6.0 (btw, also in 7.5.0):
"rawquerystring":"text_en:\"I will think about it\"",
"querystring":"text_en:\"I will think about it\"",
"parsedquery":"SpanNearQuery(spanNear([spanOr([spanNear([text_en:i, text_en:will_think], 0, true), spanNear([text_en:i_will, text_en:think], 0, true)]), text_en:about_it], 0, true))", ...
I've found the culprit.
Sharing it to support future googlers.
There was a mistake in the field configuration in schema.xml file.
In the "query" analyzer, instead of "CommonGramsFilterFactory", it should be "CommonGramsQueryFilterFactory".

Is Solr SuggestComponent able to return shingles instead of whole field values?

I use solr 5.0.0 and want to create an autocomplete functionality generating suggestions from the word-grams (or shingles) of my documents.
The problem is that in return of a suggest-query I only get complete "terms" of the search field which can be extremly long.
CURRENT PROBLEM:
Input:"so"
Suggestions:
"......extremly long text son long text continuing......"
"......next long text solar next text continuing......"
GOAL:
Input: "so"
Suggestions with shingles:
"son"
"solar"
"solar test"
etc
<searchComponent name="suggest" class="solr.SuggestComponent"
enable="${solr.suggester.enabled:true}" >
<lst name="suggester">
<str name="name">mySuggester</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">title_and_description_suggest</str>
<str name="weightField">price</str>
<str name="suggestAnalyzerFieldType">autocomplete</str>
<str name="queryAnalyzerFieldType">autocomplete</str>
<str name="buildOnCommit">true</str>
</lst>
schema.xml:
<fieldType name="autocomplete" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_de.txt" format="snowball"/>
<filter class="solr.ShingleFilterFactory" maxShingleSize="2" outputUnigrams="true" outputUnigramsIfNoShingles="true"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
I want to return max 3 words as autocomplete term. Is this possible with the SuggestComponent or how would you do it? No matter what I try I always receive the complete field value of matching documents.
Is that expected behaviour or what did I do wrong?
Many thanks in advance
In schema.xml define fieldType as follows:
<fieldType name="text_autocomplete" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ShingleFilterFactory" minShingleSize="2" maxShingleSize="5"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
In schema.xml define your field as follows:
<field name="example_field" type="text_autocomplete" indexed="true" stored="true"/>
Write your query as follows:
query?q=*&
rows=0&
facet=true&
facet.field=example_field&
facet.limit=-1&
wt=json&
indent=true&
facet.prefix=so
In the facet.prefix field, specify the term being searched for which you want suggestions ('so', in this example). If you need less than 5 words in the suggestion, reduce maxShingleSize in the fieldType definition accordingly. By default, you will get the results in decreasing order of their frequency of occurrence.

Resources