SOLR backup status request not found - solr

Question

I have been trying to figure out a way that I know when the SOLR backup is done and its status. We have a lot of collections that we are trying to backup. The request has an error
status={state=notfound,msg=Did not find [requestId123] in any tasks queue}
When I looked at the SOLR source code, I realized that the status is reported from the request status in the overseer queue i.e. COMPLETED,FAILED,RUNNING,SUBMITTED is based on the overseer queue. When the request in not found in the overseer queue or when the queue is cleared then we get this error.
My question is there any other way to get the SOLR backup status reliably.
Thanks

Answer 1

Taking backup
I am not sure how you are running the process for a backup (nor where you can see that error). My assumption is that you are checking logs (because it looks like a similar message which will appear in logs).
Additionally you did not mention which solr version you are using. I will elaborate the answer bellow for 8.9 (but any version which supports v2 AND v1 api should work similar).
If you want to run backup asynchronously you can use following:
curl -X POST http://localhost:8983/api/collections -H 'Content-Type: application/json' -d '
{
"backup-collection": {
"name": "openaccess-v26-backup",
"collection": "openaccess-v26",
"location": "/var/solr/mounted-efs-backup",
"async": "1000"
}
}
'
This will start async process for a backup with track id 1000.
Checking action status
You can use following to check the status of the process:
curl 'http://localhost:8983/solr/admin/collections?action=REQUESTSTATUS&requestid=1000'
This will return response like this:
{
"responseHeader":{
"status":0,
"QTime":3},
"status":{
"state":"running",
"msg":"found [1000] in running tasks"}}
Additionally, this is the way to check all of the actions (not only backup). For example, using the same way you can check status of RESTORE action if you are restoring your backup into the solr collection.
Listing backups
It seems that relevant info as well can be to try to list backups from time to time and see is the backup within your list (if above approach is not working for you).
Please make a note that I am not 100% sure is there a possibility for backup to be listed if its not completed, but based on my testing and pure empirical approach seems that this is not the case.
So, if I start backup, and try to execute an api which is going to give me a list of all of the backups, I will get empty list for example:
curl -X POST http://localhost:8983/v2/collections/backups -H 'Content-Type: application/json' -d '
{
"list-backups" : {
"name": "openaccess-v26-backup",
"location": "/var/solr/mounted-efs-backup"
}
}'
{
"responseHeader":{
"status":0,
"QTime":165},
"backups":[]
}
}
However, if you execute this after a while (when backup is completed), the response will be in a following format:
{
"responseHeader":{
"status":0,
"QTime":14},
"collection":"openaccess-v26",
"backups":[{
"indexFileCount":0,
"indexSizeMB":0.0,
"shardBackupIds":{
"shard2":"md_shard2_0.json",
"shard3":"md_shard3_0.json",
"shard1":"md_shard1_0.json"},
"collection.configName":"openaccess-v26",
"backupId":0,
"collectionAlias":"openaccess-v26",
"startTime":"2022-07-05T08:34:53.703175Z",
"indexVersion":"8.9.0"}]}
This kind of approach works fine for the 8.9 version of solr im using with apiv2.
I was able to restore and use backups without any kind of issues after they are listed.