I am using SOLR to generate suggestions. I have given a float field named length (tfloat) as the weightField in solrconfig.xml. But in my suggestions, this value does not come up in the suggestion response and shows as zero. I was expecting the suggestions to be sorted by weight and the weight to hold the value of length which represents the string length of the suggestions. I am using solr 6.2.1
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">mySuggester</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="indexPath">suggester_infix_dir</str>
<str name="payloadField">payload</str>
<str name="weightField">lengthval</str>-->
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">value</str>
<str name="suggestAnalyzerFieldType">text_general</str>
<str name="buildOnStartup">true</str>
<str name="buildOnCommit">true</str>
<str name="highlight">false</str>
<requestHandler name="/suggesthandler" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.count">900</str>
<str name="suggest.dictionary">mySuggester</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
Hello it started working , not sure what was wrong
Related
I am having a lock issue when trying to initialize a Solr core:
SolrCore Initialization Failures:
MySolrCore: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: C:\solr-6.0.1\server\solr\MySolrCore\data\analyzingInfixSuggesterIndexDir\write.lock
I am trying to implement multiple spell checkers on this core and I get this lock error.
Some interesting notes from trying different approaches:
I tried extending the writeLockTimeout to 30 seconds and that did nothing.
None of the spellcheckIndexDir folders are getting created. I added them in hopes of fixing the lock issue. I even tried creating them manually, and no files were added to them.
If I comment out one of the components and handlers, the other one works.
If I comment out the spellecheckers that use the AnalyzingInfixLookupFactory then both of the spellcheckers that use the WFSTLookupFactory and both of the handlers work. However, the spellcheckIndexDir is not created.
How do I fix the lock issue? I think that if I can get all of the spellcheckIndexDir specified to work that the indexes will be located in different folders and there won't be a lock. Thanks for any help!
Here's my Solr config search components and handlers for the two fields.
<searchComponent name="NameSuggest" class="solr.SpellCheckComponent">
<lst name="spellchecker">
<str name="name">NameFST</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.fst.WFSTLookupFactory</str>
<str name="field">NameFST</str>
<int name="minQueryLength">3</int>
<str name="buildOnCommit">true</str>
<str name="buildOnOptimize">true</str>
<str name="suggestAnalyzerFieldType">suggest_phrase</str>
<str name="spellcheckIndexDir">./NameFSTSuggestIndexDir</str>
</lst>
<lst name="spellchecker">
<str name="name">NameAI</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingInfixLookupFactory</str>
<str name="field">NameAI</str>
<int name="minQueryLength">3</int>
<str name="buildOnCommit">true</str>
<str name="buildOnOptimize">true</str>
<str name="suggestAnalyzerFieldType">suggest_phrase</str>
<str name="spellcheckIndexDir">./NameAISuggestIndexDir</str>
</lst>
<str name="queryAnalyzerFieldType">suggest_phrase</str>
</searchComponent>
<requestHandler class="org.apache.solr.handler.component.SearchHandler" name="/NameSuggest" startup="lazy">
<lst name="defaults">
<str name="spellcheck">true</str>
<str name="spellcheck.dictionary">NameFST</str>
<str name="spellcheck.dictionary">NameAI</str>
<str name="spellcheck.onlyMorePopular">true</str>
<str name="spellcheck.count">10</str>
<str name="spellcheck.collate">false</str>
<str name="spellcheck.extendedResults">true</str>
</lst>
<arr name="components">
<str>NameSuggest</str>
</arr>
</requestHandler>
<searchComponent name="DescriptionSuggest" class="solr.SpellCheckComponent">
<lst name="spellchecker">
<str name="name">DescriptionFST</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.fst.WFSTLookupFactory</str>
<str name="field">DescriptionFST</str>
<int name="minQueryLength">3</int>
<str name="buildOnCommit">true</str>
<str name="buildOnOptimize">true</str>
<str name="suggestAnalyzerFieldType">suggest_phrase</str>
<str name="spellcheckIndexDir">./DescriptionFSTSuggestIndexDir</str>
</lst>
<lst name="spellchecker">
<str name="name">DescriptionAI</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingInfixLookupFactory</str>
<str name="field">DescriptionAI</str>
<int name="minQueryLength">3</int>
<str name="buildOnCommit">true</str>
<str name="buildOnOptimize">true</str>
<str name="suggestAnalyzerFieldType">suggest_phrase</str>
<str name="spellcheckIndexDir">./DescriptionAISuggestIndexDir</str>
</lst>
<str name="queryAnalyzerFieldType">suggest_phrase</str>
</searchComponent>
<requestHandler class="org.apache.solr.handler.component.SearchHandler" name="/DescriptionSuggest" startup="lazy">
<lst name="defaults">
<str name="spellcheck">true</str>
<str name="spellcheck.dictionary">DescriptionFST</str>
<str name="spellcheck.dictionary">DescriptionAI</str>
<str name="spellcheck.onlyMorePopular">true</str>
<str name="spellcheck.count">10</str>
<str name="spellcheck.collate">false</str>
<str name="spellcheck.extendedResults">true</str>
</lst>
<arr name="components">
<str>DescriptionSuggest</str>
</arr>
</requestHandler>
Instead of spellcheckIndexDir I needed to use indexPath. It's working now.
<!-- <str name="spellcheckIndexDir">./NameAISuggestIndexDir</str> -->
<str name="indexPath">./NameAISuggestIndexDir</str>
I am using the Solr Suggester component in Solr 5.5 with a lot of address data. My Machine has allotted 20Gb RAM for solr and the machine has 32GB RAM in total.
I have an address book core with the following vitals -
"numDocs"=153242074
"segmentCount"=34
"size"=30.29 GB
My solrconfig.xml looks something like this -
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">mySuggester1</str>
<str name="lookupImpl">FuzzyLookupFactory</str>
<str name="storeDir">suggester_fuzzy_dir</str>
<!-- Substitute these for the two above for another "flavor"
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name=?indexPath?>suggester_infix_dir</str>
-->
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">site_address</str>
<str name="suggestAnalyzerFieldType">suggestType</str>
<str name="payloadField">property_metadata</str>
<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>
</lst>
<lst name="suggester">
<str name="name">mySuggester2</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="indexPath">suggester_infix_dir</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">site_address_other</str>
<str name="suggestAnalyzerFieldType">suggestType</str>
<str name="payloadField">property_metadata</str>
<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>
</lst>
</searchComponent>
The handler is defined like so -
<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy" >
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.count">10</str>
<str name="suggest.dictionary">mySuggester1</str>
<str name="suggest.dictionary">mySuggester2</str>
<str name="suggest.collate">false</str>
<str name="echoParams">explicit</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
Problem Statement
Every time I try to build the suggest index using the suggest.build=true url parameter, I end up with an OutOfMemory error. I have no clue how I can make this work with the current setup. Can anyone explain why this is happening? And how can I fix this issue?
On the mailing list, the critical info is that the OOME is due to java heap space.
This means that you either need to increase your heap size or reduce the amount of heap that Solr needs. It is not always possible to reduce the heap requirements, but here are some ideas:
https://wiki.apache.org/solr/SolrPerformanceProblems#Reducing_heap_requirements
I have the following problem with Solr's suggester: it's returning many times the same value, and all weights are zero. I would hope it would return each value only one time, with weights according to the frequency.
This is my config:
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">infixSuggester</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="indexPath">infix_suggestions</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">title_suggest</str>
<str name="suggestAnalyzerFieldType">phrase_suggest</str>
<str name="buildOnStartup">true</str>
<str name="buildOnCommit">true</str>
</lst>
</searchComponent>
<requestHandler name="/suggest" class="solr.SearchHandler">
<lst name="defaults">
<str name="suggest.dictionary">infixSuggester</str>
<str name="suggest.onlyMorePopular">true</str>
<str name="suggest">true</str>
<str name="suggest.count">500</str>
<str name="suggest.collate">true</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
when I call for example suggest?q=test, I get something like:
"suggestions":[{
"term":"This is a test suggestion",
"weight":0,
"payload":""},
{
"term":"This is a test suggestion",
"weight":0,
"payload":""},
{
"term":"Another test example",
"weight":0,
"payload":""}]
So I have to do processing in my client to get only the most popular completions. I would hope to have something closer to:
"suggestions":[{
"term":"This is a test suggestion",
"weight":2,
"payload":""},
{
"term":"Another test example",
"weight":1,
"payload":""}]
Is that possible at all?
Thanks!
Yann
You could use BlendedInfixLookupFactory suggester instead, its an advanced version of AnalyzingInfixLookupFactory, providing you with other features and removes the duplicates... Win-Win!
I am trying to implement the auto suggest of solr this is the changes that I made in solrconfig.xml file
<requestHandler class="org.apache.solr.handler.component.SearchHandler" name="/suggest">
<lst name="defaults">
<str name="spellcheck">true</str>
<str name="spellcheck.dictionary">suggest</str>
<str name="spellcheck.onlyMorePopular">true</str>
<str name="spellcheck.count">5</str>
<str name="spellcheck.collate">true</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
<searchComponent class="solr.SpellCheckComponent" name="suggest">
<lst name="spellchecker">
<str name="name">suggest</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookupFactory</str>
<str name="field">displayName</str> <!-- the indexed field to derive suggestions from -->
<float name="threshold">0.005</float>
<str name="buildOnCommit">true</str>
</lst>
</searchComponent>
when I try to query with sample input as 'p'
http://localhost:8983/solr/food/suggest?q=p&wt=json&indent=true
it returns 5 words
"pizza", "potato", "pasta", "protein", "premium"
but in the displayName field I got words like paneer , palak etc which is not showing up why is it so?
Can you added the following to your configuration and run the below query. Don't forget to reload the solr core after putting these changes.
<str name="suggestAnalyzerFieldType">string</str>
<str name="storeDir">suggester_fuzzy_dir</str>
http://localhost:8983/solr/food/suggest?suggest=true&suggest.build=true&suggest.dictionary=suggest&wt=json&suggest.q=p&suggest.count=10
All:
I am using solr 4.10 default configuration now, I wonder if any one can give me a working configuration for suggester part? I tried the example in Solr in Action, but it does not work(I tried this /suggest handler in admin page, keeping other options default only specifying a keyword in q ).
<requestHandler name="/suggest"
class="org.apache.solr.handler.component.SearchHandler">
<lst name="defaults">
<str name="echoParams">none</str>
<str name="wt">json</str>
<str name="indent">false</str>
<str name="spellcheck">true</str>
<str name="spellcheck.dictionary">suggestDictionary</str>
<str name="spellcheck.onlyMorePopular">true</str>
<str name="spellcheck.count">5</str>
<str name="spellcheck.collate">false</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
<searchComponent class="solr.SpellCheckComponent" name="suggest">
<lst name="spellchecker">
<str name="name">suggestDictionary</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.fst
.FSTLookupFactory</str>
<str name="field">suggest</str>
<float name="threshold">0.</float>
<str name="buildOnCommit">true</str>
</lst>
</searchComponent>
Thanks