Dynamically field selection for Solr Suggestion (Spellcheck) multiple term query - solr

I am using the solr suggestion component with the following configuration:
schema.xml
<fieldType name="textSpell" class="solr.TextField">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<field name="image_memo" type="text_general"/>
<field name="username" type="text_general"/>
<field name="image_memo" type="text_general"/>
<field name="image_text" type="text_general"/>
<!-- More fields included here -->
<field name="spell" type="textSpell" indexed="true" stored="true" multiValued="true"/>
<copyField source="*" dest="spell"/>
solrconfig.xml
<searchComponent class="solr.SpellCheckComponent" name="suggest">
<lst name="spellchecker">
<str name="name">suggest</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str>
<str name="field">spell</str>
<str name="buildOnCommit">true</str>
</lst>
</searchComponent>
<requestHandler class="org.apache.solr.handler.component.SearchHandler"
name="/suggest">
<lst name="defaults">
<str name="spellcheck">true</str>
<str name="spellcheck.dictionary">suggest</str>
<str name="spellcheck.onlyMorePopular">true</str>
<str name="spellcheck.count">6</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.maxCollations">6</str>
<str name="spellcheck.maxCollationTries">1000</str>
<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.collateParam.mm">100%</str>
</lst>
<arr name="components">
<str>suggest</str>
<str>query</str>
</arr>
</requestHandler>
As you can see there is a field spell wich i am using for a suggestion queries.
This works great even for multiple term queries.
But what I need is to search on selected fields only.
So for example I want valid suggestions only for the fields image_memo and username
The user can dynamicly add and remove fields to search.
I know that I could do something like this:
q=(image_memo:*search* OR image_username:*search*)
But this is is slowing down dramtically if you got a lot of fields and a multiple term query.
Example: Searching in Field memo, username, field, field1 and field2 for term, term1 and term2.
((memo:term OR username:term OR field:term OR field1:term OR
field2:term) AND (memo:term1 OR username:term1 OR field:term1
OR field1:term1 OR field2:term1) AND (memo:term2 OR
username:term2 OR field:term2 OR field1:term2 OR
field2:term2))
Is there any way to dynamically select the spell fields. Or is there a way that I can search for specific fields only in a multivalued field
I am using Apach Solr 4 Alpha.

All you need to do is use Dismax or eDismax. SpellcheckComponent automatically runs every suggestion using your query params.
So, you have to query like this:
/suggest?q={!dismax}term1 term2 term3&qf=memo username field field1 field2
or
/suggest?q=term1 term2 term3&defType=dismax&qf=memo username field field1 field2
You can implement your custom queryparser if you don't want to use (e)dismax.

Related

Solr - Suggester custom field not detected & Multiple fields cannot work

Goal
Trying to implement an auto-suggester in Solr. Fields to extra suggestions from are title and content fields.
Progress thus far
I followed the official Solr guide to implement the feature, however, was stuck for a long time, as it was complaining that the custom field suggestType was not defined.
After a long time of trying I decided to add the field type to managed-schema.xml instead of schema.xml and it worked!
Thus far, it only worked when I based the suggestion field off content, however, we would like to use 2 fields to base suggestions of which is title and content.
Steps followed
1) Add custom field type in managed-schema xml:
<fieldType name="suggestType" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="[^a-zA-Z0-9]" replacement=" " />
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
2) Add custom field which uses custom field type in schema.xml:
<field name="suggestText" type="suggestType" stored="true" indexed="true" />
3) Add 'suggest' handler in solr-config.xml:
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">fuzzySuggester</str>
<str name="lookupImpl">FuzzyLookupFactory</str>
<str name="storeDir">fuzzy_suggestions</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">suggestText</str>
<str name="suggestAnalyzerFieldType">suggestType</str>
<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>
</lst>
</searchComponent>
<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy" >
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.dictionary">analyzingSuggester</str>
<str name="suggest.onlyMorePopular">true</str>
<str name="suggest.count">10</str>
<str name="suggest.collate">true</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
4) Copy both fields 'title' and 'content' to 'suggestText' in schema.xml:
<copyField source="title" dest="suggestField"/>
<copyField source="content" dest="suggestField"/>
Questions
Why does it only work when I add the custom field type to managed-schema.xml instead of schema.xml? From my understanding, managed-schema.xml should not be manually edited.
No results seem to appear when after I map both title and content field to the custom field textSuggest. I would like to know what I am missing.
Thanks.
It seems like you have a typo in your copy-field definition. The "dest" attribute is suggestField but the field you created earlier is called suggestText.

Solr diffrence between analysis results and indexed values

I want to use transformations by replacements on my extracted/indexed values of few copied fields with Solr.
In my schema.xml, I have the source_file field who contains a value. I copy this value into the collection field. Then I use PatternReplaceCharFilterFactory (regex / replacement) to modify this value as I want :
<field name="source_file" type="string" indexed="true" stored="true" multiValued="false" docValues="false"/>
<field name="collection" type="collectionType" indexed="true" stored="true" multiValued="false"/>
<copyField source="source_file" dest="collection"/>
<fieldType name="collectionType" class="solr.TextField" stored="true" indexed="true">
<analyzer type="index">
<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="A" replacement="B"/>
<tokenizer class="solr.KeywordTokenizerFactory"/>
</analyzer>
<analyzer type="query">
<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="A" replacement="B"/>
<tokenizer class="solr.KeywordTokenizerFactory"/>
</analyzer>
</fieldType>
EDIT 1
In Solr webapp, when I use the Analysis interface, I see that the pattern replacement works for Index & Query forms.
My source_file value is modified by the setted value. Which means that the definition of my schema seems correct. A becomes B.
But, when I use a Solr Query, I see that the pattern replacement not works. No change is taken into account (I always have the value A).
http://xxx:898x/solr/discovery_collection/select?fl=collection&indent=on&q=*:*&wt=json
The result shows that the value of collection field is not modify by the replacement pattern when I use a query.
I've done all my configurations into schema.xml. Nothing into solrconfig.xml.
Would I have forgotten a configuration (into solrconfig.xml ?) to force the indexing of modified fields?
EDIT 2
My request handlers in solrconf.xml :
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">text</str>
<str name="defType">edismax</str>
<str name="timeAllowed">30000</str>
<bool name="preferLocalShards">false</bool>
</lst>
</requestHandler>
<requestHandler name="/query" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="wt">json</str>
<str name="indent">true</str>
<str name="df">text</str>
<int name="rows">10</int>
<str name="defType">edismax</str>
<str name="timeAllowed">30000</str>
</lst>
</requestHandler>
<requestHandler name="/update/extract"
startup="lazy"
class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
<str name="lowernames">true</str>
<str name="uprefix">ignored_</str>
<!-- capture link hrefs but ignore div attributes -->
<str name="captureAttr">true</str>
<str name="fmap.a">links</str>
<str name="fmap.div">ignored_</str>
</lst>
</requestHandler>
EDIT 3
I've just seen that it works. But in fact the value "B" is stored/indexed and I can search it by query, but the value "B" is not displayed in the json result of the query (it displays the value "A").
Do have I to use custuom updateRequestProcessorChain and requestHandler in solrconfig.xml? And add a custom class in lib/ to do the pattern transformation?

Returning single word from Solr Suggester

I am developing a web application, and am using Solr as search engine. I would like to add autocomplete functionality. To do this, I have added the Suggester component, and configured a separate field for it. This works ok.
The problem is that Suggester returns the whole value of the field. For example, if the name of an article is "A newsworthy item" and I search for "new", it will return the whole "A newsworthy item", where I would like it to just return "newsworthy". In other words, return the individual word tokens.
The schema looks like this:
<fieldType name="text_autocomplete" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<field name="term" type="text_autocomplete" indexed="true" stored="true" multiValued="false" />
<field name="weight" type="float" indexed="true" stored="true" />
<copyField source="name" dest="term"/>
The values are copied into the "term" field. The Solr config:
<!-- Search component -->
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">suggester</str>
<str name="lookupImpl">AnalyzingLookupFactory</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">term</str>
<str name="weightField">weight</str>
<str name="suggestAnalyzerFieldType">text_autocomplete</str>
<str name="buildOnStartup">false</str>
</lst>
</searchComponent>
<!-- Search handler -->
<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="suggest">true</str>
<str name="suggest.count">10</str>
<str name="suggest.dictionary">suggester</str>
<str name="suggest.build">true</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
Can anyone suggest a schema and/or configuration that will make the Suggester return a single word?
Instead of solr.SuggestComponent try making use of solr.SpellCheckComponent.
As SuggestComponent is meant to suggest the full phrase.
You can look into the details of solr.SpellCheckComponent over here.
http://wiki.apache.org/solr/SpellCheckComponent
For you quick reference, you can try with this.
<searchComponent name="suggest" class="solr.SpellCheckComponent">
<lst name="spellchecker">
<str name="name">suggest</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookupFactory</str>
<str name="distanceMeasure">org.apache.lucene.search.spell.JaroWinklerDistance</str>
<str name="field">term</str>
<str name="accuracy">0.7</str>
<float name="thresholdTokenFrequency">.0001</float>
</lst>
</searchComponent>

Solr autocomplete on words not in beginning of string

I have some docs in Solr that contains information about books. One of the fields is author, defined as:
<field name="author" type="text_general" indexed="true" stored="true"/>
This is an example of a doc:
<doc>
<str name="id">db04</str>
<str name="isbn">0596529325</str>
<str name="author">Toby Segaran</str>
<str name="category">Computers/Programming/Information Retrieval/Machine Learning</str>
<arr name="title">
<str>Programming Collective Intelligence</str>
</arr>
<int name="yearpub">2007</int>
<date name="pubdate">2007-07-28T00:00:01Z</date>
</doc>
I'm trying to create a autocomplete system using Solr 4.2. So far it worked well, if I search for to it returns me Toby Segaran as the result.
But in our website many people searches for Segaran for instance and I was wondering if is it possible to somehow suggest Toby Segaran when this happens.
So far this is the schema.xml I'm using:
<field name="author_suggest" type="text_auto" indexed="true" stored="true" multiValued="false"/>
<copyField source="author" dest="author_suggest"/>
<fieldType class="solr.TextField" name="text_auto">
<analyzer>
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Basically the field author is processed and copied to author_suggest.
In solrconfig.xml, these were created:
<searchComponent class="solr.SpellCheckComponent" name="suggest">
<lst name="spellchecker">
<str name="name">suggest</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookupFactory</str>
<str name="field">author_suggest</str>
<float name="threshold">0.005</float>
<str name="buildOnCommit">true</str>
</lst>
</searchComponent>
<requestHandler class="org.apache.solr.handler.component.SearchHandler" name="/suggest">
<lst name="defaults">
<str name="spellcheck">true</str>
<str name="spellcheck.dictionary">suggest</str>
<str name="spellcheck.onlyMorePopular">true</str>
<str name="spellcheck.count">6</str>
<str name="spellcheck.collate">false</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
So is this possible to somehow make suggestions based on words that are not exactly at the beginning of the phrase using the suggester from Solr?
If you need more information please let me know.
Thanks in advance

error in suggester component in solr

I am working with solr auto complete functionality,I am using solr 4.50 to build my application, and I am following this link as a reference. My suggest component is something like this
<searchComponent class="solr.SpellCheckComponent" name="suggest">
<lst name="spellchecker">
<str name="name">suggest</str>
<str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str>
<str name="storeDir">suggest</str>
<str name="field">autocomplete_text</str>
<bool name="exactMatchFirst">true</bool>
<float name="threshold">0.005</float>
<str name="buildOnCommit">true</str>
<str name="buildOnOptimize">true</str>
</lst>
<lst name="spellchecker">
<str name="name">jarowinkler</str>
<str name="field">lowerfilt</str>
<str name="distanceMeasure">org.apache.lucene.search.spell.JaroWinklerDistance</str>
<str name="spellcheckIndexDir">spellchecker</str>
</lst>
<str name="queryAnalyzerFieldType">edgytext</str>
</searchComponent>
but, I am getting the following error
org.apache.solr.spelling.suggest.Suggester – Loading stored lookup data failed
java.io.FileNotFoundException: /home/anurag/Downloads/solr-4.4.0/example/solr/collection1/data/suggest/tst.dat (No such file or directory)
It says that some file are missing but the solr wiki suggester component says it supports these lookupImpls --
<str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str>
<!-- Alternatives to lookupImpl:
org.apache.solr.spelling.suggest.fst.FSTLookup [finite state automaton]
org.apache.solr.spelling.suggest.fst.WFSTLookupFactory [weighted finite state automaton]
org.apache.solr.spelling.suggest.jaspell.JaspellLookup [default, jaspell-based]
org.apache.solr.spelling.suggest.tst.TSTLookup [ternary trees]
-->
Dont know what I am doing wrong..... Any help will be deeply appreciated
I was able to get the autosuggest functionality working by using the Solr Term Component
Add term components in your solrconfig.xml like this
<searchComponent name="terms" class="solr.TermsComponent"/>
<!-- A request handler for demonstrating the terms component -->
<requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<bool name="terms">true</bool>
<bool name="distrib">false</bool>
</lst>
<arr name="components">
<str>terms</str>
</arr>
</requestHandler>
define a field type for your autosuggest text in schema.xml
<fieldType name="edgytext" class="solr.TextField" >
<analyzer>
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
add fields in schema.xml like this
<field name="name" type="edgytext" indexed="true" stored="true" />
<field name="autocomplete_text" type="edgytext" indexed="true" stored="false" multiValued="true" omitNorms="true" omitTermFreqAndPositions="false" />
<copyField source="name" dest="autocomplete_text"/>
Now the most important step... Remove all the folders from your index directory
(can be found in solrconfig.xml ,.. look for <dataDir> tag)
Restart the solr. and reindex your data. You will se new folders created in your index directory.
You can check the auto suggest working by hitting the url -
http://127.0.0.1:8983/solr/your_core/terms?terms.fl=autocomplete_text&omitHeader=true&terms.limit=20&terms.sort=index&terms.regex=(.*)your_query(.*)

Resources