Using Solr version 7.3.1
Starting with 3 nodes:
I have created a collection like this:
wget "localhost:8983/solr/admin/collections?action=CREATE&autoAddReplicas=true&collection.configName=my_col_config&maxShardsPerNode=1&name=my_col&numShards=1&replicationFactor=3&router.name=compositeId&wt=json" -O /dev/null
In this way I have a replica on each node.
GOAL:
Each shard should add a replica to new nodes joining the cluster.
When a node are shoot down. It should just go away.
Only one replica for each shard on each node.
I know that it should be possible with the new AutoScalling API but I am having a hard time finding the right syntax. The API is very new and all I can find is the documentation. Its not bad but I am missing some more examples.
This is how its looks today. There are many small shard each with a replication factor that match the numbers of nodes. Right now there are 3 nodes.
This video was uploaded yesterday (2018-06-13) and around 30 min. into the video there is an example of the Solr.HttpTriggerListener that can be used to call any kind of service, for example an AWS Lamda to add new nodes.
The short answer is that your goals are not not achievable today (till Solr 7.4).
The NodeAddedTrigger only moves replicas from other nodes to the new node in an attempt to balance the cluster. It does not support adding new replicas. I have opened SOLR-12715 to add this feature.
Similarly, the NodeLostTrigger adds new replicas on other nodes to replace the ones on the lost node. It, too, has no support for merely deleting replicas from cluster state. I have opened SOLR-12716 to address that issue. I hope to release both the enhancements in Solr 7.5.
As for the third goal:
Only one replica for each shard on each node.
To achieve this, a policy rule given in the "Limit Replica Placement" example should suffice. However, looking at the screenshot you've posted, you actually mean a (collection,shard) pair which is unsupported today. You'd need a policy rule like the following (following does not work because collection:#EACH is not supported):
{"replica": "<2", "collection": "#EACH", "shard": "#EACH", "node": "#ANY"}
I have opened SOLR-12717 to add this feature.
Thank you for these excellent use-cases. I'll recommend asking questions such as these on the solr-user mailing list because not a lot of Solr developers frequent Stackoverflow. I could only find this question because it was posted on the docker-solr project.
Related
There are indexes of some solr cores which I convert them from solr4 to solr6 but in solr standalone mode. so they don't have the "version" field that solrcolud require.
Here now I want to migrate to solrcloud 6 and I need to put them under cluster. Because the version field dose not exist there in these indexes when I put them Under a solrcloud leader core on the data directory the replicas in the shard didn't update as I saw. so I decided to read them by lucene, get each doc fields, add them to a solrdoc and then put them doc by doc in solrcloud. But cause there are fields that not stored in these indexes so all fields that exist here in these indexes don't move there.
At the end it seems there is no way for me than re-indexing.
I appreciate if there is any better idea or solutions that can help me migrate more easily.
If there is any chance to reindex, just do so, it's going to be the best in the end (you have to deal with two separate issues: a) migrate from 4.X to 6.0 and b)from standalone to SolrCloud...it's going to be messy).
If you cannot reindex:
are all your fields stored OR have docValues=true? If so, you can get the original contents of your docs. Read them and index them with solrj or with some script.
if not, and you have a version field: try to manually put the index in Solrcloud. Not straighforward, but possible.
if you don't have a version field, I think it is impossible to put the index as is in Solrcloud (although some post on the net make you think it is). You could try to write some lucene code to add version field to all docs (with values that make sense), but this should be the very last resort.
We have a SolrCloud setup with 20 shards, each with only 1 replica, served on 8 servers.
After a server went down we are left with 16 shards, which means that some of the compositeId hash ranges aren't hosted by any cores. Somehow the shards/cores didn't come back after the server came up again. I can see the server in /live_nodes.
But all is not bad: The data in the collection is volatile with a TTS of 30 minutes, and we have a failover in place that tries a new random compositeId whenever an "add" operation fails.
My question is: Is it possible to recreate the missing shards or do I have to delete and create the collection from scratch?
I know which hash ranges are are missing, but the CREATESHARD API call doesn't support shards with the 'compositeId' router. And I cannot use SPLITSHARD since it only divides the original shard's hash.
(We use Solr 5.4.0 and can't upgrade before 6.1 is released, ref. SOLR-8940)
So, I asked the same question at the solr-user mailing list. The answer I got suggests using zkNavigator and editing state.json, adding the missing shard(s) with the correct parameters. An ADDREPLICA API call is needed afterwards.
I haven't gotten around to trying it yet, but I will do if the problem reoccurs.
We have standalone solr servers which are master and slave. Also have a full indexer job nightly. Generally, when job executed successful everything is alright. But last days, we noticed that indexer node has different document number with searching node. So, expected productions are not available in our production system. That's why we had to restart nodes and start replication manually, then problem went away. We need to prevent to occur this problem again. What do you suggest us to check or where should i look at? Indeed i think that essential error about the issue is: "SEVERE: No files to download for index generation"
Regards
I saw from the FAQ that a DSE node can be reprovisioned from RT mode to Hadoop mode. Is something similar supported with DSE Search and DSE Spark? I have an existing 6-node DSE Search cluster. I want to test DSE Spark but I have very limited time left for development so if possible, I'd like to skip the bootstrap process by simply restarting my cluster as an Analytics DC instead of adding new nodes in a separate DC.
UPDATE:
I tried to find an answer on my own. These are the closest that I found:
http://www.datastax.com/wp-content/uploads/2012/03/WP-DataStax-WhatsNewDSE2.pdf
http://www.datastax.com/doc-source/pdf/dse20.pdf
These documents are for a very old release of DSE. Both documents say that only RT and Analytics node can be re-provisioned. The second document even explicitly says that a Solr node cannot be re-provisioned. Unfortunately, there is no mention about re-provisioning in more recent documentations.
Can anybody confirm whether this is still true with DSE 4.5.1? (preferably with a link to a reference)
I also saw this forum thread which explains why the section about re-provisioning was removed in recent documentations. However, in my case, I plan to re-provision all of my Search nodes as Analytics node (in contrast to re-provisioning only a subset), and the re-provisioning would only be temporary
Yes you can do that. Just start it using 'dse Cassandra -k'
I am using solr 4.5. After several tests I have noticed a lot of dead (non existing) replicas are shown in my SolrCloud graph as gone (black). Is there any way to force my solr to forget about this gone replicas?
I think that manually modifying /clusterstate.json node in zookeeper might help but did not try it yet.
The simplest way I found is in fact editing /clusterstate.json in zookeeper, and removing dead replicas info from it.
I don't know if there is any way to do some sort of global cleanup... but:
There is an API to remove some specific replica:
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-DeleteaReplica
As well as for removing the (INACTIVE) shard with all it's replica's (4.4+):
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-DeleteaShard
And, if this is something related to production and not only for testing purpose - you may also want to look at this upcoming change from 4.6 related to registering the replica that was previously removed - https://issues.apache.org/jira/browse/SOLR-5311