How to use Solr Suggester ContextField with boolean field - solr

I am using Solr 6.0.0
I am tring to filter out unwanted suggestions from Solr Suggester. In my Solr database I have all my products
My products all have a boolean field "ShowOnSite". Products that are ready for sale have this value set to true. Products not yet ready have it set to false.
When I try to filter the suggested results from the suggester using this boolean field, I always get 0 results, even though I have plenty of products ready to be shown.
My Products looks somewhat like this like this:
<field name="id" type="string" indexed="true" stored="true" required="true"/>
<field name="Name" type="string" indexed="true" stored="true"/>
<field name="ShowOnSite" type="boolean" indexed="true" stored="true" />
<field name="text_autocomplete" type="textSuggest" indexed="true" stored="true"/>
The textSuggest fieldType has the following configuration:
<fieldType class="solr.TextField" name="textSuggest" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
My suggester looks like this
<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.count">20</str>
<str name="wt">json</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">default</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="highlight">true</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">text_autocomplete</str>
<str name="weightField">InStock</str>
<str name="contextField">ShowOnSite</str>
<str name="suggestAnalyzerFieldType">textSuggest</str>
<str name="buildOnStartup">true</str>
</lst>
</searchComponent>
My query looks like this:
/suggest?suggest.q={querystring}&suggest.cfq=true
Expected
I receive only the products that has "ShowOnSite" == true
Actual
I receive 0 proucts from the suggester
I have tried other configurations aswell. By using not true I get all products:
/suggest?suggest.q={querystring}&suggest.cfq=-true
I have also tried to add the field name in the cfq. This yields 0 products:
/suggest?suggest.q={querystring}&suggest.cfq=ShowOnSite:true
EDIT1
I have also tried using either 0 or 1 for false and true respectively. These do not work either

Initial guess is that this is caused by the boolean type of the field, since no analysis happens as far as I know for the values used by the cfq.
Make a secondary field as a string field and store the false or true value verbatim in that field - and use that for filtering instead.

As suggested by MatsLindh. Use a text field instead.
The easiest way is to just copy that field:
Add this to the managed-schema file of your index (in Solr):
<field name="THE_FIELD_TO_BE_USED_BY_THE_SUGGESTER" type="text_general" indexed="true" stored="true" multiValued="false"/>
<copyField source="YOUR_BOOLEAN_FIELD" dest="THE_FIELD_TO_BE_USED_BY_THE_SUGGESTER" maxChars="30000" />

Related

Solr Suggester - Store Lookup build failed

I've exhausted my search efforts as to why this isn't working. I believe I'm following the documentation correctly found at https://cwiki.apache.org/confluence/display/solr/Suggester
However, every time I attempt to build the suggester, I receive the error "SolrSuggester - Store Lookup build failed." in the logs. I can see it creating the directory for the store correctly on disk, however, there is no data within the file.
I've also tried removing the line <str name="storeDir">fuzzy_dir</str>. If I do this and try building, I don't receive the error in the logs, however, I still receive no results.
Can anyone see what I may be doing wrong?
I'm using Solr 6.5.0.
Here is what I have in my schema.xml:
<field name="name" type="text_general" indexed="true" stored="true" required="true" multiValued="false" />
<field name="term" type="suggestType" indexed="true" stored="true" />
<copyField source="name" dest="term" />
<fieldType name="suggestType" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Here is what I have in my solrconfig.xml:
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">fuzzySuggester</str>
<str name="lookupImpl">FuzzyLookupFactory</str>
<str name="storeDir">fuzzy_dir</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">term</str>
<str name="suggestAnalyzerFieldType">suggestType</str>
<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>
</lst>
</searchComponent>
<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy" >
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.dictionary">fuzzySuggester</str>
<str name="suggest.count">5</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
This is how I'm executing the build:
http://localhost:8983/solr/my_core/suggest?suggest.build=true
you might be missing this in the fuzzySuggester:
<str name="weightField">WEIGHT</str>
even if the docs say it's an optional param, I think it might be what is messing with you. If you don't have a good field that you can use, you can just declare one like this:
<field name="WEIGHT" type="tfloat" indexed="true" stored="true" multiValued="false" />
and just don't bother putting any data into it.
Try giving suggester.dictionary=fuzzySuggester in the query.
http://localhost:8983/solr/my_core/suggest?suggest.build=true&suggester.dictionary=fuzzySuggester
After endless hours of scouring the internet and attempting suggestions provided by others on this post, I've come to the conclusion something in my solrconfig.xml or schema.xml file was corrupt.
My fix was to create a completely new core and migrate the pieces I was using in solrconfig.xml and schema.xml to get it to work. Unfortunately I don't have a better answer, but this is what I had to do in order to solve the problem.

SOLR 6.4.1 Suggester is stubbornly case-sensitive, how to make case-insensitive?

I've tried everything under the sun (well it is called solr after all) to make solr Suggest case-insensitive, but it stubbornly continues to be case-sensitive.
This returns a suggestion of Mexican:
http://localhost:8983/solr/mycollection/autocomplete?suggest.q=Mex
This returns 0 results:
http://localhost:8983/solr/mycollection/autocomplete?suggest.q=mex
To further diagnose I tried a lower case /select search against my suggestions field, which successfully returned docs containing "Mexican":
http://localhost:8983/solr/mycollection/select?q=suggestions:mex*
But no such luck using lowercase with the Suggester. It's as though my <filter class="solr.LowerCaseFilterFactory"/> has no effect when used by the Suggester.
I of course did a full config upload, collection reload, data re-index, and suggester rebuild before testing. I'm on SOLR 6.4.1 running in cloud mode. Any ideas? Diagnostic tips?
schema.xml
<fieldType name="textSuggest" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<field name="recipe" type="text_general" indexed="true" stored="true" multiValued="false" />
<field name="suggestions" type="textSuggest" indexed="true" stored="true" multiValued="true" />
<copyField source="recipe" dest="suggestions"/>
solrconfig.xml
<searchComponent class="solr.SuggestComponent" name="suggest">
<lst name="suggester">
<str name="name">foodsuggester</str>
<str name="lookupImpl">WFSTLookupFactory</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">suggestions</str>
<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>
<str name="storeDir">suggester_wfst_dir</str>
<str name="suggestAnalyzerFieldType">textSuggest</str>
</lst>
</searchComponent>
<requestHandler name="/autocomplete" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.dictionary">foodsuggester</str>
<str name="suggest.count">10</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
The WFSTLookupFactory apparently does not take the suggestAnalyzerFieldType parameter and it is ignored. You could use the AnalyzingLookupFactory, which will analyze the text according to the suggestAnalyzerFieldType. So if you only want the lower case to be analyzed in the suggester you can use the suggestAnalzerFieldType, and indicate that you want to use the suggestText field type for analysis through the suggestAnalyzerFieldType.
It seems the WFSTLookupFactory lookup implmentation is case sensitive.
You can use FuzzyLookupFactory, if you don't have any specific reason for using WFSTLookupFactory.
<str name="lookupImpl">FuzzyLookupFactory</str>

How to get Suggestions in Solr 5.3.0

I am trying to implement auto complete feature using Solr 5.3.0
solrconfig.xml looks like this
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">default</str>
<str name="lookupImpl">FuzzyLookupFactory</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">suggest_ngram</str>
<str name="weightField">price</str>
<str name="suggestAnalyzerFieldType">text_suggest_ngram</str>
<str name="buildOnStartup">true</str>
</lst>
</searchComponent>
<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy" >
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.count">10</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
managed-schema looks like this:
<fieldType name="text_suggest_ngram" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" maxGramSize="10" minGramSize="2" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<field name="suggest_ngram" type="text_suggest_ngram" indexed="true" stored="false"/>
<field name="name" type="string" multiValued="false" indexed="true" stored="true"/>
<field name="price" type="tlong" multiValued="false" indexed="true" stored="true"/>
<copyField source="name" dest="suggest_ngram"/>
Now when I use the analyzer from the admin panel of Solr, I can see the indexed ngrams. And it successfully points out the match.
However when I use the query:
http://localhost:8983/solr/products/suggest?suggest=true&suggest.build=true&wt=json&suggest.q=Jind
I get 0 suggestions.
The response is here:
https://api.myjson.com/bins/47r3i
There exists a value "Jindal Panther" for the name key in one of the docs.
Moreover, I have found that if I create a dummy copyfield "suggest" with type as "String", with source as "name", any suggestion that works fine on "name" will not work on "suggest". Can this be any misconfiguration of copyfield to enable suggestions?
Any help would be appreciated.
Thanks in advance.
EDIT:
Got the solution. See the accepted answer and its comments below.
There is a blog that I encountered that beautifully explains Suggesters. It is definitely worth reading for a newbie to Solr Search.
https://lucidworks.com/blog/2015/03/04/solr-suggester/
The field on which you want to configure the suggester should be store=true. It need not to be indexed. The suggester configuration will build a dictionary according to the provide configuration in the suggestComponet. The name field have stored as true where as suggest_ngram is not. You need to update the schema configuration like this:
<field name="suggest_ngram" type="text_suggest_ngram" indexed="false" stored="true"/>
Also you need to provide the parameter suggest.dictionary, the dictionary you are using for suggestions. For you it is names as default.
http://localhost:8983/solr/products/suggest?suggest=true&
suggest.build=true&
wt=json&
suggest.dictionary=default&
suggest.q=Jind
OR you can provide the dictionary configuration in requestHandler of /suggest:
<str name="suggest.dictionary">default</str>

integrating solr autosuggest functionality error

I am trying to integrate auto suggest functionality of solr in my project. I use this as my starting point. I changed my searched fields accordingly.
my schema.xml
<field name="name" type="text_suggest" indexed="true" stored="true"/>
<field name="manu" type="text_suggest" indexed="true" stored="true"/>
<field name="popularity" type="int" indexed="true" stored="true" />
<!-- A variant of textsuggest which only matches from the very left edge -->
<copyField source="name" dest="textnge"/>
<field name="textnge" type="autocomplete_edge" indexed="true" stored="false" />
<!-- A variant of name which matches from the left edge of all terms (implicit truncation) -->
<copyField source="name" dest="textng"/>
<field name="textng" type="autocomplete_ngram" indexed="true" stored="false" omitNorms="true" omitTermFreqAndPositions="true" />
My request handler in solrconfig.xml
<requestHandler class="solr.SearchHandler" name="/ac" default="true" >
<lst name="defaults">
<str name="defType">edismax</str>
<str name="rows">10</str>
<str name="fl">*,score</str>
<str name="qf">name^50 manu^20.0 textng^50.0</str>
<str name="pf">textnge^50.0</str>
<str name="bf">product(log(sum(popularity,1)),100)^20</str>
<str name="debugQuery">false</str>
</lst>
</requestHandler>
The problem is that my "/ac" handler is acting more like "/select" handler. When I type "moni" I am getting nothing. But when I type "monitor", its returning me the documents containing monitor in them.
I have been trying this for whole day and nothing seems to work. Any help will be deeply appreciated
Well when you look for "moni" in your query, you are actually specifically saying that you're looking for the "moni" keyword. Try looking for multiterms keywrods by adding "*", such as q=moni*.
You can also look in other fieldType analyser like autocomplete_edge (q=textnge:mori) or autocomplete_ngram (q=textng:mori) for more data.
I think you need to specify search component in solarconfig.xml like below
<searchComponent class="solr.SpellCheckComponent" name="ac">
<lst name="spellchecker">
<str name="name">suggest</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookupFactory</str>
<str name="field">yourfieldname</str> <!-- the indexed field to derive suggestions from -->
<float name="threshold">0.005</float>
<str name="buildOnCommit">true</str>
</lst>
</searchComponent>

Solr full refresh doesn't clear the index

I'm having an issue where Solr won't clear the index during a full import.
All of the servers run Solr 3.4, the configuration is as vanilla as it can be.
I tried this on our development environment and on an instance on my own computer, and received similar results.
The schema is rather simple, these are the salient points:
<schema name="System" version="1.4">
...
</types>
<fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true" />
<fieldType name="long" class="solr.TrieLongField" precisionStep="0" omitNorms="true" positionIncrementGap="0" />
<fieldType name="date" class="solr.TrieDateField" omitNorms="true" precisionStep="0" positionIncrementGap="0" />
<fieldType name="documentKey" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
</analyzer>
</fieldType>
</types>
<fields>
<field name="document_id" type="documentKey" indexed="true" stored="true" required="true" />
<field name="entity_id" type="long" indexed="true" stored="true" required="true" />
<field name="name" type="string" indexed="true" stored="true" required="true" />
<field name="entity_type" type="string" indexed="true" stored="true" required="false" />
<field name="Timestamp" type="date" indexed="true" stored="true" default="NOW" multiValued="false"/>
</fields>
</schema>
Of note:
- The document_id field is calculated in the materialized view which is used to populate the index, and is a combination of other fields not in this index, but is indipendent of the entity_id. It's unique.
- The entity_id field is the key of a couple of tables, and for the same document_id it can change wildly between a refresh and another.
Before a full refresh, if I query the index as such:
http://localhost:8080/qq-solr/system/select/?rows=10&q=document_id:%22French_Polynesia/Huahine~4034376%22
I get:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">5</int>
<lst name="params">
<str name="indent">true</str>
<str name="q">document_id:"French_Polynesia/Huahine~4034376"</str>
<str name="rows">10</str>
</lst>
</lst>
<result name="response" numFound="1" start="0">
<doc>
<date name="Timestamp">2012-03-08T09:47:26.335Z</date>
<str name="document_id">French_Polynesia/Huahine~4034376</str>
<long name="entity_id">22902728</long>
<str name="name">Huahine</str>
<str name="type">LOCATION</str>
</doc>
</result>
</response>
Then I refresh:
http://localhost:8080/qq-solr/system/dataimport?command=full-import&clean=true&commit=true&optimize=true
(I know the clean, commit, and optimize are redundant, but I used them just to make sure) and after a while I get the message that everything is a-ok.
Then I query the index again:
http://localhost:8080/qq-solr/system/select/?rows=10&q=document_id:%22French_Polynesia/Huahine~4034376%22
And I get:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">5</int>
<lst name="params">
<str name="indent">true</str>
<str name="q">document_id:"French_Polynesia/Huahine~4034376"</str>
<str name="rows">10</str>
</lst>
</lst>
<result name="response" numFound="1" start="0">
<doc>
<date name="Timestamp">2012-03-09T08:31:07.317Z</date>
<str name="document_id">French_Polynesia/Huahine~4034376</str>
<long name="entity_id">22902728</long>
<str name="name">Huahine</str>
<str name="type">LOCATION</str>
</doc>
</result>
</response>
But in the database the entity_id is different!
I see that the Timestamp has been updated, so that record has been touched, but why is the old value being retained?
I would run your DataImportHandler (DIH) process through the Interactive Development Mode so that you can assure that your database query is retrieving the entity_id that you are expecting. Because the timestamp on the solr entry is being updated, your DIH process is running, but I am guessing the cause for this lies in the way the data is being retrieved.
Any time I'm doing an operation like this with Solr, I always manually clear the index first using curl to be 100% sure its wiped. Here is a tutorial: http://www.alphadevx.com/a/365-Clearing-a-Solr-search-index

Resources