I want to configure solr cloud with more than 4 nodes, mostly 6-8 nodes so that my data gets distributed among all the nodes evenly. But I am not able to configure more than 4 nodes. Can anyone guide me on how I can configure more than 4 nodes in solr cloud with zookeeper being used externally?
Related
We recently upgraded from Solr 6.6 to Solr 7.7 (cluster with 3 nodes) and are facing a peculiar issue. We index around 9-10 million docs every monday to Solr as part of a batch process and this process used to take around 50-55 mins to complete with Solr6.6.
After migration to Solr 7.7 with all configuration (solrconfig.xml and schema) exactly same as existing collection on Solr 6.6 we observed the same process is taking 5-6 hrs to index the similar amount of load.
On checking solr logs we found that most of the update requests (>90%) went to 1 of the nodes out of 3.
The solr cluster is fronted by a GCP load balancer which is configured to have a round robin policy.
We are using compositeId as router with no prefix based routing.
Exampe doc: {id:AB_111_1_1, qty:10.0, name:111, part:1, date:2020-06-29T00:00:00Z }
Solr7 cluster is enabled with auth plugin which was not enabled on Solr 6 cluster. There was no other change from application/process side for this behaviour. We are not using any solr client (solrs or solrj) to send the update/query requests to solr. It is a normal Http Connection to load balancer with auth enabled.
This behavior degrades with time, and indexing becomes much more slower. We observed similar behaviour when we used solr data import handler to copy the documents from solr 6.6 to solr 7.7. Initially the index speed was around 2K docs per sec but in 10-15 mins this used to start to get impacted and speed reduced.
Verify that apache Tika is the correct version and that the Java memory is allocated correctly.
Monitor your operation with htop
The solution to the above problem was upgrading the infrastructure on which solr was hosted. After upgrading CPU + Memory, the performance was better than Solr 6.
This was found out by trial method with different configurations.
We need to move our solr cloud cluster from one cloud vendor to another, the cluster is composed of 8 shards with 2 replica factor spread among 8 servers with roughly a total of 500GB worth of data.
I wonder what are the common approaches to migrate the cluster but specially its data with the less impact in availability and performance etc..
I was thinking in some sort of initial dump copy to then synchronize them catching up the diff (which could be huge) after keeping them in sync just switch whenever everything is ready from the other side.
Is that something doable? what tools should/could I use?
Thanks!
You have multiple choices depending on your existing setup and Solr version:
As mentioned earlier, make use of backup and restore APIs from Collections API
If you have Solr 6 and above, I would recommend exploring the option of CDCR, which is Solr's native Cross Data Centre Replication.
Reindexing onto the new cluster and then leverage Solr Collection Aliasing to change your application end points to the target provider upon the completion of reindexing
I'm currently using Solr Cloud 6.1, the following behavior can also be observed until 7.0.
I'm trying to create a Solr collection with 5 shards and a replication factor of 2. I have 5 physical servers. Normally, this would distribute all 10 replicas evenly among the available servers.
But, when starting Solr Cloud with a -h (hostname) param to give every Solr instance an individual, but constant hostname, this doesn't work any more. The distribution then looks like this:
solr-0:
wikipedia_shard1_replica1 wikipedia_shard2_replica1 wikipedia_shard3_replica2 wikipedia_shard4_replica1 wikipedia_shard4_replica2
solr-1:
solr-2:
wikipedia_shard3_replica1 wikipedia_shard5_replica1 wikipedia_shard5_replica2
solr-3:
wikipedia_shard1_replica2
solr-4:
wikipedia_shard2_replica2
I tried using Rule-based Replica Placement, but the rules seem to be ignored.
I need to use hostnames, because Solr runs in a Kubernetes cluster, where IP adresses change frequently and Solr won't find it's cores after a container restart. I first suspected a newer Solr version to be the cause of this, but I narrowed it down to the hostname problem.
Is there any solution for this?
The solution was actually quite simple (but not really documented):
When creating a Service in OpenShift/Kubernetes, all matching Pods get backed by a load balancer. When all Solr instances get assigned an unique hostname, this hostnames would all resolve to one single IP address (that of the load balancer).
Solr somehow can't deal with that and fails to distribute its shards evenly.
The solution is to use headless services from Kubernetes. Headless services aren't backed by a load balancer and therefore every hostname resolves to an unique IP address.
we are using solr 4.8.1 cloud (1shard with four nodes)for our environment. There is an inconsistency in latest data coming into three nodes.
Example, we are commiting data to solr1 and that data is getting replicated to solr3 after 5 minutes, where as solr 2 and solr 4 gets it immediately.
Also this behaviour changes when we restart the cluster
so if X was lagging after restart it may not lag again
Any ideas about the issue?
Hi I'm developing rails project with sunspot solr and configuring Solr Cloud.
My environment: rails 3.2.1, ruby 2.1.2, sunspot 2.1.0, Solr 4.1.6.
Why SolrCloud: I need more stable system - oftentimes search server goes on maintenance and web application stop working on production. So, I think about how to make 2 identical search servers instead of one, to make system more stable: if one server will be down, other will continue working.
I cannot find any good turtorial with simple, easy to understand and described in details turtorial...
I'm trying to set up SolrCloud on two servers, but I do not fully understand how it is working inside:
synchronize data between two servers (is it automatic action?)
balances search requests between two servers
when one server suddenly stop working other should become a master (is it automatic action?)
is there SolrCloud features other than listed?
Read more about SolrCloud here..! https://wiki.apache.org/solr/SolrCloud
Couple of inputs from my experience.
If your application just reads data from SOLR and does not write to SOLR(in real time but you index using an ETL or so) then you can just go for Master Slave hierarchy.
Define one Master :- Point all writes to here. If this master is down you will no longer be able to index the data
Create 2(or more) Slaves :- This is an feature from SOLR and it will take care of synchronizing data from the master based on the interval we specify(Say every 20 seconds)
Create a load balancer based out of slaves and point your application to read data from load balancer.
Pros:
With above setup, you don't have high availability for Master(Data writes) but you will have high availability for data until the last slave goes down.
Cons:
Assume one slave went down and you bought it back after an hour, this slave will be behind the other slaves by one hour. So its manual task to check for data consistency among other slaves before adding back to ELB.
How about SolrCloud?
No Master here, so you can achieve high availability for Writes too
No need to worry about data inconsistency as I described above, SolrCloud architecture will take care of that.
What Suits Best for you.
Define a external Zookeeper with 3 nodes Quorom
Define at least 2 SOLR severs.
Split your Current index to 2 shards (by default each shard will reside one each in 2 solr nodes defined in step #2
Define replica as 2 (This will create replica for shards in each nodes)
Define an LB to point to above solr nodes.
Point your Solr input as well as application to point to this LB.
By above setup, you can sustain fail over for either nodes.
Let me know if you need more info on this.
Regards,
Aneesh N
-Let us learn together.