I have trouble with Apache Solr now.
I wonder if there is any difference with 2 query:
status:1 AND province:D500 AND district:014 AND precinct:008 AND name:*THÔN 22*
I have parsedquery_toString": "+status:1 +province:D500 +district:014 +precinct:008 +name:*thôn _text:22*"
and with the second one:
status:1 AND name:*THÔN 22* AND province:D500 AND district:014 AND precinct:008
I have
"parsedquery_toString": "+status:1 +name:*thôn +_text:22* +province:D500 +district:014 +precinct:008",
What makes the parsequery_toString to be so different with just the changes of orders of query?
It could be the result of the space in *THÔN 22*, and how it changes the query. If it's a single search term instead of two, try quoting it and see if you get the same results then.
Related
I have a set of test results in my mongodb database. Each document in the database contains version information, test data, date, test run information etc...
The version is broken up in the document and stored as individual values. For example: { VER_MAJOR : "0", VER_MINOR : "2", VER_REVISION : "3", VER_PATCH : "20}
My application wants the ability to specify a specific version and grab the document as well as the previous N documents based on the version.
For example:
If version = 0.2.3.20 and n = 5 then the result would return documents with version 0.2.3.20, 0.2.3.19, 0.2.3.18, 0.2.3.17, 0.2.3.16, 0.2.3.15
The solutions that come to my mind is:
Create a new database that contains documents with version information and is sorted. Which can be used to obtain the previous N version's which can be used to obtain the corresponding N documents in the test results database.
Perform the sorting in the test results database itself like in number 1. Though if the test results database is large, this will take a very long time. Also consider inserting in order every time.
Creating another database like in option 1 doesn't seem like the right way. But sorting the test results database seems like there will be lots of overhead, am I mistaken that I should be worried about option 2 producing lots of overhead? I have the impression I'd have to query the entire database then sort it on application side. Querying the entire database seems like overkill...
db.collection_name.find().sort([Paramaters for sorting])
You are quite correct that querying and sorting the entire data set would be very excessive. I probably went overboard on this, but I tried to break everything down in detail below.
Terminology
First thing first, a couple terminology nitpicks. I think you're using the term Database when you mean to use the word Collection. Differentiating between these two concepts will help with navigating documentation and allow for a better understanding of MongoDB.
Collections and Sorting
Second, it is important to understand that documents in a Collection have no inherent ordering. The order in which documents are returned to your app is only applied when retrieving documents from the Collection, such as when specifying .sort() on a query. This means we won't need to copy all of the documents to some other collection; we just need to query the data so that only the desired data is returned in the order we want.
Query
Now to the fun part. The query will look like the following:
db.test_results.find({
"VER_MAJOR" : "0",
"VER_MINOR" : "2",
"VER_REVISION" : "3",
"VER_PATCH" : { "$lte" : 20 }
}).sort({
"VER_PATCH" : -1
}).limit(N)
Our query has a direct match on the three leading version fields to limit results to only those values, i.e. the specific version "0.2.3". A range $lte filter is applied on VER_PATCH since we will want more than a single patch revision.
We then sort results by VER_PATCH to return results descending by the patch version. Finally, the limit operator is used to restrict the number of documents being returned.
Index
We're not done yet! Remember how you said that querying the entire collection and sorting it on the app side felt like overkill? Well, the database would doing exactly that if an index did not exist for this query.
You should follow the equality-sort-match rule when determining the order of fields in an index. In this case, this would give us the index:
{ "VER_MAJOR" : 1, "VER_MINOR" : 1, "VER_REVISION" : 1, "VER_PATCH" : 1 }
Creating this index will allow the query to complete by scanning only the results it would return, while avoiding an in-memory sort. More information can be found here.
I am trying to upgrade my Solr 4.x version to 5.2.1 Solrcloud implementation. I had written following code to get all the results from Sorl query which works well in Solr single instance mode.
SolrQuery query = new SolrQuery();
query.setQuery("*:*");
query.addSort("agent_status", ORDER.desc);
query.addFilterQuery("account_id:\"" + accountId + "\"");
query.set("rows", Integer.MAX_VALUE);
But code will not work well in SolrCloud implemenation.It throws following exception.
2015-08-14 16:44:45,648 ERROR [solr.core.SolrCore] - [http-8080-8] : java.lang.NegativeArraySizeException
at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:58)
at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:39)
at org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113)
at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:972)
at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:750)
at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:729)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:388)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
I found that it is failing because of query.set("rows", Integer.MAX_VALUE) statement.People suggested me to use pagination.
But, I can not afford doing pagination as there will be too many changes at UI side.
There is one more way where I can first query with some small number & get total number of documents using response.getResults().getNumFound() method & try setting that value to setRows method.But this approach will increase one more call to server.
Is there any other way I can solve this problem?
You can always set your rows to be a large value that would encompass your results. Integer.MAX_VALUE will not work due to the size limits of Java Arrays (see here) and the Lucene Priority Queue (see lines 42 - 58).
Solr-534 requested to have essentially what your asking for; there is some good conversation about why and why-not such a feature would be good.
A better question might be how many documents can the UI hold without becoming unusable? However many documents that is, would be a good value for your query to return.
I have a situation where I have to search a document in Solr with multiple OR keywords. Now the number of keywords may lead up to 5000 which is resulting in a awfully large query with 5000 OR conditions. This is resulting in the Solr server to hang. Is there any other way I can design the query to work. Short sample of the query is given below
tweet_id:337931022601699328 OR 337931064293081089 OR 337931089538584576 OR 337931098761871361 OR 337931138851016704 OR 337931143099854848 OR 337931160082591745 OR 337931163857453056 OR 337931230819516416 OR 337931239996665857 OR 337931287518126080 OR 337931322850951168 OR 337931325648535553 OR 337931331398934528 OR 337931413057830912 OR 337931442363441152 OR 337931448629731329 OR 337931453344129025 OR 337931465016877056 OR 337931482066726912 OR 337931514388029442 OR 337931533149155328 OR 337931645527130114 OR 337931704935256064 OR 337931784459268096 OR 337931845545103360 OR 337931889086185472 OR 337931892668108801 OR 337931963983855617 OR 337932154212319233 OR 337932176454721536 OR 337932193198374912 OR 337932229659459584 OR 337932437290090496 OR 337932436807749632 OR 337932436828725250 OR 337932437449474048 OR 337932448518250496 OR 337932458832035843 OR 337932458634915840 OR 337932458278387712 OR 337932474246119425 OR 337932476209041409 OR 337932477408620544 OR 337932480478842880 OR 337932478775959554 OR 337932480566931456 OR 337932478763376640 OR 337932481841999872 OR 337932479337992192 OR 337932479296045057 OR 337932479333797889 OR 337932484614434816 OR 337932484606038017 OR 337932482777317376 OR 337932484664758272 OR 337932482785718273 OR 337932484589273088 OR 337932487399444481 OR 337932489031032833 OR 337932489114923008 OR 337932486573166592 OR 337932490704560130 OR 337932489144270848 OR 337932488762601472 OR 337932492097069056 OR 337932497780355072 OR 337932498900230144 OR 337932499722321921 OR 337932514431729665 OR 337932561806409731 OR 337932567284154368 OR 337932567300935680 OR 337932574603214848 OR 337932571134533632 OR 337932574674518016 OR 337932575484026881 OR 337932578206121984 OR 337932582215892994 OR 337932586653454336 OR 337932584917024768 OR 337932592986865664 OR 337932597017587712 ....
I intend to facet the result based on a few fields.
I'm not sure whether this solution would help you or not, but tried something for your problem.
Whatever the query you provide to Solr, first it parses that query to it's understandable format. Then Solr executes that for result. You have to do some calculations before querying to Solr. Let's take the following scenario to solve your use case.
Suppose You have total 5000 tweet_id. You have to do an OR query on around 4000 tweet_id. In this type of scenario, it's better to query on other (5000-4000=1000) 1000 tweet_id with negation AND query. So, your query will have less values passed.
So, try querying with rest of the tweet_id with negation AND query instead of OR query.
If I were you, I'd create a new field denoting this custom_list_id .. Whenever you generate a new list, index the new data then query by the list I'd.
By the address:
http://myhost.com:8983/solr/#/collection1/query
I can not find the possibility for inputing GROUP queries, like:
group=true&group.field=geohash1_st
Where is fields for that type of query?
That is what I have:
P.S: I have solr 4 version.
P.S2: I can type my query into URL of my browser, but it is not convenient, because the query is long! So the question is to make that using prepared input fields...
The place you'd put those is labelled "Raw Query Parameters" ... but it's not in your screenshot. This probably means that you're not using a new enough version of Solr 4. I can confirm that this is in the admin UI as of version 4.2.1, and it's definitely in the latest version, 4.6.0.
You disappeared from #solr before I could answer there.
I faced same issue.
I have done like this :
We need to send parameter on : Raw Query Parameters
stats.facet=Status&stats.field=ItemPrice&stats=on&wt=json
stats.facet=Status - It means group by Status.
stats.field=ItemPrice - It means Sum,min,max,sd.. macro aggregation will be done on ItemPrice.
stats=on It enable the stats
Result will contain Stats like below :
"stats":{
"stats_fields":{
"ItemPrice":{
"min":1.0,
"max":1399.99,
"count":8,
"missing":0,
"sum":3147.9399999999987,
"sumOfSquares":3063146.2605999997,
"mean":393.49249999999984,
"stddev":510.5257066635194,
"facets":{
"Status":{
"6":{
"min":1.0,
"max":1399.99,
"count":8,
"missing":0,
"sum":3147.9399999999987,
"sumOfSquares":3063146.2605999997,
"mean":393.49249999999984,
"stddev":510.5257066635194}}}}}}}
I executed some query like "Address:Jack*". It show numFound = 5214 and display 100 documents in results page(I changed default display results from 10 to 100).
How can I get all documents.
I remember myself doing &rows=2147483647
2,147,483,647 is integer's maximum value. I recall using a number bigger than that once and having a NumberFormatException because it couldn't be parsed into an int. I don't know if they use Long nowadays, but 2 billion rows is normally more than enough.
Small note:
Be careful if you are planning to do this in production. If you do a query like * : * and your index is big, you could transferring a couple of gigabytes in that query.
If you know you won't have many docs, go ahead and use integer's max value.
On the other hand, if you are doing a one-time script and just need to dump all results (for example document ID's) then this approach is valid, if you don't mind waiting 3-5 minutes for a query to return.
Don't use &rows=2147483647
Don't use Integer.MAX_VALUE(2147483647) as value of rows in production. This will heavily slow down your query even if you have a small resultset, because solr preallocates a queue in this size. see https://issues.apache.org/jira/browse/SOLR-7580
I strongly suggest to use Exporting Result Sets
It’s possible to export fully sorted result sets using a special rank query parser and response writer specifically designed to work together to handle scenarios that involve sorting and exporting millions of records.
Or I suggest to use Deep Paging.
Simple Pagination is a easy thing when you have few documents to read and all you have to do is play with start and rows parameters. But this is not a feasible way when you have many documents, I mean hundreds of thousands or even millions.
This is the kind of thing that could bring your Solr server to their knees.
For typical applications displaying search results to a human user,
this tends to not be much of an issue since most users don’t care
about drilling down past the first handful of pages of search results
— but for automated systems that want to crunch data about all of the
documents matching a query, it can be seriously prohibitive.
This means that if you have a website and are paging search results, a real user do not go so further but consider on the other hand what can happen if a spider or a scraper try to read all the website pages.
Now we are talking of Deep Paging.
I’ll suggest to read this amazing post:
https://lucidworks.com/post/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/
And take a look at this document page:
https://solr.apache.org/guide/pagination-of-results.html
And here is an example that try to explain how to paginate using the cursors.
SolrQuery solrQuery = new SolrQuery();
solrQuery.setRows(500);
solrQuery.setQuery("*:*");
solrQuery.addSort("id", ORDER.asc); // Pay attention to this line
String cursorMark = CursorMarkParams.CURSOR_MARK_START;
boolean done = false;
while (!done) {
solrQuery.set(CursorMarkParams.CURSOR_MARK_PARAM, cursorMark);
QueryResponse rsp = solrClient.query(solrQuery);
String nextCursorMark = rsp.getNextCursorMark();
for (SolrDocument d : rsp.getResults()) {
...
}
if (cursorMark.equals(nextCursorMark)) {
done = true;
}
cursorMark = nextCursorMark;
}
Returning all the results is never a good option as It would be very slow in performance.
Can you mention your use case ?
Also, Solr rows parameter helps you to tune the number of the results to be returned.
However, I don't think there is a way to tune rows to return all results. It doesn't take a -1 as value.
So you would need to set a high value for all the results to be returned.
What you should do is to first create a SolrQuery shown below and set the number of documents you want to fetch in a batch.
int lastResult=0; //this is for processing the future batch
String query = "id:[ lastResult TO *]"; // just considering id for the sake of simplicity
SolrQuery solrQuery = new SolrQuery(query).setRows(500); //setRows will set the required batch, you can change this to whatever size you want.
SolrDocumentList results = solrClient.query(solrQuery).getResults(); //execute this statement
Here I am considering an example of search by id, you can replace it with any of your parameter to search upon.
The "lastResult" is the variable you can change after execution of the first 500 records(500 is the batch size) and set it to the last id got from the results.
This will help you execute the next batch starting with last result from previous batch.
Hope this helps. Shoot up a comment below if you need any clarification.
For selecting all documents in dismax/edismax via Solarium php client, the normal query syntax : does not work. To select all documents set the default query value in solarium query to empty string. This is required as the default query in Solarium is :. Also set the alternative query to :. Dismax/eDismax normal query syntax does not support :, but the alternative query syntax does.
For more details following book can be referred
http://www.packtpub.com/apache-solr-php-integration/book
As the other answers pointed out, you can configure the rows to be max integer to yield back all the results for a query.
I would recommend though to use Solr feature of pagination, and build a function that will return for you all the results using the cursorMark API. The gist of it is you set the cursorMark parameter to '*', you set the page size(rows parameter), and on each result you'll get a cursorMark for the next page, so you execute the same query only with the cursorMark given from the last result. This way you'll have more flexibility on how much of the results you want back, in a much more performant way.
The way I dealt with the problem is by running the query twice:
// Start with your (usually small) default page size
solrQuery.setRows(50);
QueryResponse response = solrResponse(query);
if (response.getResults().getNumFound() > 50) {
solrQuery.setRows(response.getResults().getNumFound());
response = solrResponse(query);
}
It makes a call twice to Solr, but gets you all matching records....with the small performance penalty.
query.setRows(Integer.MAX_VALUE);
works for me!!