I need to create pivot data by applying filter of two fields i.e city and date range.
Able to filter data by cities and date range but filter is not applying on stats field.
Following is the Solr query that i'm using :
select?&fq={!tag=f1}city:(%221000%20OAKS%22)&facet=true&facet.query=true&facet.query={!tag=queryOne}datadate:[2015-01-01%20TO%202015-12-31]&facet.query={!tag=queryTwo}datadate:[2014-01-01%20TO%202014-12-31]&stats=true&stats.field={!tag=a1%20sum=true%20key=charge1}charge&stats.field={!tag=a2%20sum=true%20key=spend1}spend&facet.pivot={!query=queryOne%20key=c1%20stats=a1}city&facet.pivot={!query=queryTwo%20stats=a2%20key=c2}city&facet=on&indent=on&wt=json&rows=0&q=:
Actual Result :
As you can see, the sum remains same (i.e 2348) regardless of what datadate i m giving in queryone & querytwo tags.
It appears that datadate filters is not having any effect on stats field.
Can anyone please explain what is happening here as i'm pretty new to this.
Thanks in advance.
The problem is that the StatsComponent & FacetComponent are not in perfect integration with each other. The following Solr Jira comprises your use-case: SOLR-6348.
But nevertheless there is work-around for your use-case via JSON Facet API. Here is facet parameter which solves your particular use-case:
{
"date_range": {
"type": "range",
"field": "datadate_dt",
"start": "2014-01-01T00:00:00Z",
"end": "2016-01-01T00:00:00Z",
"gap": "%2B1YEAR",
"facet": {
"cities_and_spends": {
"type": "terms",
"field": "city_s",
"facet": {"sum_of_spend": "sum(spend_d)"}
},
"cities_and_charges": {
"type": "terms",
"field": "city_s",
"facet": {"sum_of_spend": "sum(charge_d)"}
}
}
}
}
Hence the overall GET request would look like:
.../select?indent=on&q=*:*&fq=city:(%221000%20OAKS%22)&rows=0&wt=json&json.facet={%20%22date_range%22:%20{%20%22type%22:%20%22range%22,%20%22field%22:%20%22datadate_dt%22,%20%22start%22:%20%222014-01-01T00:00:00Z%22,%20%22end%22:%20%222016-01-01T00:00:00Z%22,%20%22gap%22:%20%22%2B1YEAR%22,%20%22facet%22:%20{%20%22cities_and_spends%22:%20{%20%22type%22:%20%22terms%22,%20%22field%22:%20%22city_s%22,%20%22facet%22:%20{%22sum_of_spend%22:%20%22sum(spend_d)%22}%20},%20%22cities_and_charges%22:%20{%20%22type%22:%20%22terms%22,%20%22field%22:%20%22city_s%22,%20%22facet%22:%20{%22sum_of_spend%22:%20%22sum(charge_d)%22}%20}%20}%20}%20}
Related
I'm having trouble creating a Solr query to be able to pull out the right documents, and am starting to wonder if what I am trying to do is even possible.
Currently on Solr 8.9 using a managed schema and every field is using a wildcard field.
Firstly what the document looks like
(changed names due to redacting internal business language):
{
"id": "COUNTY:1",
"county_name_s": "Hertfordshire",
"coordinates_s": {
"id": "COUNTY:1COORDINATES:!",
"lat_s": "54.238948",
"long_s": "54.238948"
},
"cities": [
{
"id": "COUNTY:1CITY:1",
"city_name_s": "St Albans",
"size": {
"id": "COUNTY:1CITY:1SIZE:1",
"sq_ft_s": "100",
"sq_meters_s": "5879"
}
},
{
"id": "COUNTY:1CITY:2",
"city_name_s": "Watford",
"size": {
"id": "COUNTY:1CITY:2SIZE:2",
"sq_ft_s": "150",
"sq_meters_s": "10000"
}
}
],
"mayor": {
"title_s": "Mrs.",
"first_name_s": "Sheila",
"last_name_s": "Smith"
}
}
And what I want to return:
{
"id": "COUNTY:1",
"county_name_s": "Hertfordshire",
"coordinates": {
"id": "COUNTY:1COORDINATES:!",
"lat_s": "54.238948",
"long_s": "54.238948"
},
"cities": [
{
"id": "COUNTY:1CITY:1",
"city_name_s": "St Albans",
"size": {
"id": "COUNTY:1CITY:1SIZE:1",
"sq_ft_s": "100",
"sq_meters_s": "5879"
}
}
],
"mayor": {
"title_s": "Mrs.",
"first_name_s": "Sheila",
"last_name_s": "Smith"
}
}
Basically my goal is to return more or less the entire thing, however with filtering out one of the cities. For example, the condition for the city would be like city_name_s:"St Albans". So it's to say that I want the parent and all children, however if the child is in that array (ie cities array), then the given field (city_name_s) must equal my defined value, or we don't want that child.
Things I've tried:
I've basically tried two approaches here:
I've tried to play around with {!child} and {!parent} to get a result that I want. Currently I can only get something from City level or the entire thing as if the filter was not there at county level.
I've tried to change values for the childFilter option, with things like:
city_name_s:"St Albans" OR (*:* NOT city_name_s:[* TO *]) to try and say 'if field exists it should be this'.
Anyhow I'm starting to run out of ideas with this; been hacking away at it for the past couple of days and not really got any closer.
Thanks in advance for any help; bashing my head against the wall currently so any suggestions are more than welcome :)
I had a similar issue in solr 9.0.0 and this solved it for me: Apache Solr Filter on Child Documents
In your case, just add fl=*,[child childFilter=city_name_s:"St Albans"]
I started Latency tracking and passed the /jobs/metrics of the REST API. Obtained the following information:
{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.e5ebb093256018a0621f548fbe118f8a.operator_subtask_index.0.latency_p75"},
{"id":"lastCheckpointExternalPath"}, {"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.55785f9edccd37ac9093dea77018f09d.operator_subtask_index.0.latency_max"}, {"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.55785f9edccd37ac9093dea77018f09d.operator_subtask_index.0.latency_p999"},
.....
Q： Why aren't the specific values displayed? How should I use the API correctly?
Thank you for your help.
See the doc.
/jobs/metrics return the metrics' id, you need query with job id and metric id:
GET taskmanagers/ABCDE/metrics?get=metric1,metric2
to get values like:
[
{
"id": "metric1",
"value": "34"
},
{
"id": "metric2",
"value": "2"
}
]
So, reading about an issue on the official documentation, I figured that we can have multivalued field as :
solr.add([
{
"id": "doc_1",
"title": "A test document",
"link": ["baidu.com", "google.com"]
}
Now, I would like to index the docs on both the title and link field, how can I do that? And what is the default field it is indexed on?
In my instance of Solr 4.10.3 I would like to index JSONs with a nested structure.
Example:
{
"id": "myDoc",
"title": "myTitle"
"nestedDoc": {
"name": "test name"
"nestedAttribute": {
"attr1": "attr1Val"
}
}
}
I am able to store it correctly through the admin interface:
/solr/#/mySchema/documents
and I'm also able to search and retrieve the document.
The problem I'm facing is that when I get the response document from my Solr search, I cannot see the nested attributes. I only see:
{
"id": "myDoc",
"title": "myTitle"
}
Is there a way to include ALL the nested fields in the returned documents?
I tried with : "fl=[child parentFilter=title:myTitle]" but it's not working (ChildDocTransformerFactory from:https://cwiki.apache.org/confluence/display/solr/Transforming+Result+Documents). Is that the right way to do it or is there any other way?
I'm using: Solr 4.10.3!!!!!!
To get returned all the nested structure, you indeed need to use ChildDocTransformerFactor. However, you first need to properly index your documents.
If you just passed your structure as it is, Solr will index them as separate documents and won't know that they're actually connected. If you want to be able to correctly query nested documents, you'll have to pre-process your data structure as described in this post or try using (modifying as needed) a pre-processing script. Unfortunately, including the latest Solr 6.0, there's no nice and smooth solution on indexing and returning nested document structures, so everything is done through "workarounds".
Particularly in your case, you'll need to transform your document structure into this:
{
"type": "parentDoc",
"id": "myDoc",
"title": "myTitle"
"_childDocuments_": [
{
"type": "nestedDoc",
"name": "test name",
"_childDocuments_" :[
{
"type": "nestedAttribute"
"attr1": "attr1Val"
}]
}]
}
Then, the following ChildDocTransformerFactor query will return you all subdocuments (btw, although it says it's available since Solr 4.9, I've actually only seen it in Solr 5.3... so you need to test):
q=title:myTitle&fl=*,[child parentFilter=type:parentDoc limit=50]
Note, although it returns all nested documents, the returned document structure will be flattend (alas!), i.e., you'll get:
{
"type": "parentDoc",
"id": "myDoc",
"title": "myTitle"
"_childDocuments_": [
{
"type": "nestedDoc",
"name": "test name"
},
{
"type": "nestedAttribute"
"attr1": "attr1Val"
}]
}
Probably, not really what you've expected but... this is the unfortunate Solr's behavior that will be fixed in a nearest future release.
You can put
q={!parent which=}
and in fl field :"fl=*,[child parentFilter=title:myTitle].
It will give you all parent field and children field of title:mytitle
I'm pretty new to Solr, I'm trying to add a multi-value field with boost values defined for each value, all defined via JSON. In other words, I'd like this to work:
[{ "id": "ID1000",
"tag": [
{ "boost": 1, "value": "A test value" },
{ "boost": 2, "value": "A boosted value" } ]
}]
I know how to do that in XML (multiple <field name = 'tag' boost = '...'>), but the JSON code above doesn't work, the server says "Error parsing JSON field value. Unexpected OBJECT_START". Has Solr a limit/bug?
PS: I fixed the originally-missing ']' and that's not the problem.
EDIT: It seems the way to go should be payloads (http://wiki.apache.org/solr/Payloads), but I couldn't make them to work on Solr (followed this: http://sujitpal.blogspot.co.uk/2011/01/payloads-with-solr.html). Leaving the question open to see if someone can further help.
Found the following sentence in the from the Solr Relevancy FAQ - Query Elevation Component section
An Index-time boost on a value of a multiValued field applies to all values for that field.
I do not think adding an individual boost to each value in the multivalued field is going to work. I know that the Xml will allow it, but I would guess that it may only apply the boost value from the last value applied to the field.
So based on that I would change the Json to the following and see if that works.
[
{
"id": "ID1000",
"tag": {
"boost": 2,
"value": [ "A test value", "A boosted value"]
}
}
]
The JSON seems to be invalid missing a closing ]
[
{
"id": "ID1000",
"tag": [
{
"boost": 1,
"value": "A test value"
},
{
"boost": 2,
"value": "A boosted value"
}
]
}
]
You hit an edge case. You can have the boosts on single values and you can have an array of values. But not one inside another (from my reading of Solr 4.1 source code)
That might be something to create as an enhancement request.
If you are generating that JSON by hand, you can try:
"tag": { "boost": 1, "value": "A test value" },
"tag": { "boost": 2, "value": "A boosted value" }
I believe Sols will merge the values then. But if you are generating it via a framework, it will most likely disallow or override multiple object property names (tag here).
The error has nothing to do with boosting.
I get the same error with a very simple json doc.
No luck solving it.
see Solr errors when trying to parse a collection: Error parsing JSON field value. Unexp ected OBJECT_START
I hit the same error message. Actually the error message was misplaced. The underlying real error was the two of the required fields as per schema.xml in solr configuration were missing in the json payload.
An error message of the kind "required parameters are missing in the document" would have been more helpful here. You might want to check if some required fields are missing in the json payload.