With the Levenshtein implementation of Lucene 4 claiming to be 100 times faster than before (http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html) I would like to do fuzzy matching of all terms in a query. The idea is that a search for 'gren hose' should be able to find the document 'green house' (I don't really care about phrases at this point, the quotes are just here to make this more readable).
I am using Lucene 4 + Solr 4. As I'm doing some pre- and post-processing there is a small wrapper servlet around Solr, the servlet is using SolrJ to eventually access Solr
I'm currently a little lost on what would be the right way to achieve this. My basic approach is to break the search query down into terms and append the tilde / fuzzy operator to each term. Thus 'gren hose' would become 'gren~ hose~' . Now the question is how to properly do this. I can see several ways:
Brute force: Assume that the terms are delimited by whitespace, so just parse the query and append a tilde before each whitespace (ie. after each term)
Two steps: Send the query to Solr with query debugging turned on. This will give me a list of query terms as parsed by Solr. I can then extract the terms from the debug output, append the tilde operator and re-run the query with the added tilde operators
Internally: Hook into the search request handler and append the tilde operator after the query has been parsed into terms
Method 1 stinks a lot, as it circumvents Solr's query parsing entirely, so I would rather not do that. Method 2 sounds quite doable if the cost of parsing the query twice is not too high. Method 3 sound just right, but I have yet to figure out where I have to hook into the processing chain.
Maybe there is a completely different way to achieve what I want to do, or maybe it's just a stupid idea on my part. Anyway, I would really appreciate a few pointers, maybe someone else has already done something like this. Thanks!
I would propose the following methods:
Implement a query handler module in your application where you can build solr query from the input user query. This way nothing changes in the SOLR side and your application has all the control on what goes into SOLR.
Implement your own query parser , you can start from Standard SOLR query parser (org.apache.solr.search.QParser) and make your changes. Your application just needs to select your custom query parser and rest your implementation should take care.
I would prefer method 1 as this makes the system completely agnostic to SOLR upgrades, any new release of Solr will not require me to update the custom qparser and you will not have to update/build and setup your custom qparser in the new version.
If you dont have any control on the app and dont want to go through the qparser route , then you can implement a Servlet filter that transforms the solr query before it is dispatched to solr request filter.
Related
Is there any parameter like (edismax or dismax or any other) that i can set for stemming to work in Solr or i need to make changes in schema.xml of Solr to implement the stemming ?
Problem is if i change schema.xml by default stemming/phoentic work which i dont want ? I am using Solr from third party application and in UI we have checkbox for stemming to check/uncheck , i pass these paramaters to Solr and get the data from Solr, i cant pass this UI parameter to SOlr, so if there is any parameter at Solr side i can pass that for stemming to work ?
Please let me know ?
Stemming is performed as part of the analysis chain, and therefor is part of how the schema for that particular field is defined.
The reason for this becomes apparent when you consider how stemming works - for stemming to make sense, the term has to be stemmed when it's being indexed, as well as when being queried.
Lucene takes your input string, runs it through your analysis chain and saves the generated tokens to its index. Giving it what are you asking will probably end up as what, are, you, ask after tokenizing by whitespace and applying stemming.
The same operation happens when querying, so if someone searches for asks, the token gets stemmed to ask - and then compared against what's in the index. If stemming hadn't taken place when indexing, you'd end up with asking in the index, and ask when querying - and that isn't a match, since the tokens aren't the same.
In your third party application the stemming option probably performs stemming inside the application before sending the content to Solr.
You can also use the Schema API to dynamically update and change field type definitions.
I am currently using Apache Solr 5.5.1 and I would like to generate fuzzy results without having to add the "~" at the end of the query. As an example if I run a query against "Fellowships" it does not find any records that contain "fellowship" without the ending "s" but if I run the query against "Fellowships~" it does find these "fellowship" records. How can I configure Solr (like solrconfig.xml) so that the "~"/fuzzy query is implied without be having to add it to the end of all queries?
You'd probably be better off using a stemmer to actually stem terms to their common format, instead of using fuzzy search to achieve the same.
There is no way to make all terms behave fuzzy, as it's part of the query syntax and not an argument by itself. If you really want to apply it as a fuzzy query, append the ~x in your middleware.
I have a set of keywords defined by client requirements stored in a SOLR field. I also have a never ending stream of sentences entering the system.
By using the sentence as the query against the keywords I am able to find those sentences that match the keywords. This is working well and I am pleased. What I have essentially done is reverse the way in which SOLR is normally used by storing the query in Solr and passing the text in as the query.
Now I would like to be able to extend the idea of having just a keyword in a field to having a more fully formed SOLR query in a field. Doing so would allow proximity searching etc. But, of course, this is where life becomes awkward. Placing SOLR query operators into a field will not work as they need to be escaped.
Does anyone know if it might be possible to use the SOLR "query" function or perhaps write a java class that would enable such functionality? Or is the idea blowing just a bit too much against the SOLR winds?
Thanks in advance.
ES has percolate for this - for Solr you'll usually index the document as a single document in a memory based core / index and then run the queries against that (which is what ES at least used to do internally, IIRC).
I would check out the percolate api with ElasticSearch. It would sure be easier using this api than having to write your own in Solr.
I am using Solrj to build queries for Solr server.
So I have some pretty short free-form texts that can contain various special characters - like Mr. John's New-Wall, "Hotels & Food".
A phrase query for text like this would not produce enough matches. So from this text I would like to extract terms for building a simple query, something like content:Mr OR content:John's OR content:Hotels OR content:Food. (It probably would be good to somehow consider the term proximity, but I have to start with something).
The field that I am searching is the default text_general field. I started with replacing some special characters with spaces and splitting them up to extract the terms. But it feels kind of redundant.
Isn't there an easier way to extract terms from text using Solrj and Solr? Basically I would like to extract terms from text similarly to how it is done by Solr when it creates its index.
I am not sure exactly what your question is, however here is a bit of info that you may find helpful:
Basically I would like to extract terms from text similarly to how it is done by Solr when it creates its index.
You can configure indexing and query field processing in your schema. I would suggest you take a look in here. This gives you a bit of flexibility to normalize your data.
So from this text I would like to extract terms for building a simple query, something like content:Mr OR content:John's OR content:Hotels OR content:Food.
This is the default way that solr queries under the hood. I would suggest you look up edismax query parser and qf and tie parameters.
Hope it helps
I would like to fire complex queries in Solr 4. If I am using Lucene, I can search using XML Query parser and get the results I need. However, I am not able to see how to use the XML Query Parser in Solr.
I need to be able to execute queries with proximity searches, booleans, wildcards, span or, phrases (although these can be handled by proximity searches).
Guidance on material on how to proceed also welcome.
Regards
Puneet
As far as I know it's still a work in progress. More info can be found at their Jira. You can of course use the normal query language, it's also capable of doing pretty complex things, for example:
"a proximity search"~2 AND *wildcards* OR "a phrase"
As you can see you can search for phrases, boolean operators (AND, OR, ...), span, proximity and wildcards. For more information about the query syntax look at the Lucene documentation. Solr also added some extra features on top of the Lucene query parser and more information about that can be found at the Solr wiki.
Solr 4.8 now has the "complexphrase" query parser built in that can construct all sorts of complex proximity queries (i.e. phrase queries with embedded boolean logic and wildcards).
you can use the query url as
http://xx.xxx.xx.xx:8983/solr/collectionname/select?indent=on&q=
{!complexphrase%20inOrder=true}"good*"&wt=json&fl=Category,keywords,ImageID