SolrCloud 6 continuously reloading core every 10 seconds - solr

Question

I'm experiencing a loop in reloading core. My current configuration is:
One VM CentOS 7.1 (16GB RAM, 4 cores)
3 istances Zookeeper 3.4.9 (port 2181, 2182, 2183)
3 istances Solr 6.5.1 (port 8983, 8501, 8502)
1 collection named MERITO divided into 2 shards (one in 8983, one in 8501), each shard replicated (both replicas in 8502).
The cluster is currently active and everything seems to work fine. Looking at the solr.log file I notice a loop in reloading cores, it keeps reloading every 10 seconds circa.
Here is the block that is repeating:
10:24:50.911 INFO (qtp575335780-18) [ ] o.a.s.c.CoreContainer Reloading SolrCore 'merito_shard1_replica2' using configuration from collection merito
10:24:50.934 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.SolrCore [[merito_shard1_replica2] ] Opening new SolrCore at [/home/user/Desktop/Solr/instance2/server/solr/merito_shard1_replica2], dataDir=[/home/user/Desktop/Solr/instance2/server/solr/merito_shard1_replica2/data/]
10:24:50.934 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer#66ea810
10:24:50.935 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
10:24:50.950 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
10:24:50.950 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
10:24:50.953 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher#5747d7f5[merito_shard1_replica2] main]
10:24:50.955 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/merito
10:24:50.956 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/merito
10:24:50.956 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
10:24:50.956 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
10:24:50.957 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
10:24:50.961 INFO (searcherExecutor-584-thread-1-processing-n:192.168.94.133:8501_solr x:merito_shard1_replica2 s:shard1 c:merito r:core_node2) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher#5747d7f5[merito_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(6.5.1):C1)))}
10:24:50.967 INFO (searcherExecutor-584-thread-1-processing-n:192.168.94.133:8501_solr x:merito_shard1_replica2 s:shard1 c:merito r:core_node2) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.QuerySenderListener QuerySenderListener done.
10:24:50.967 INFO (searcherExecutor-584-thread-1-processing-n:192.168.94.133:8501_solr x:merito_shard1_replica2 s:shard1 c:merito r:core_node2) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
10:24:50.968 INFO (searcherExecutor-584-thread-1-processing-n:192.168.94.133:8501_solr x:merito_shard1_replica2 s:shard1 c:merito r:core_node2) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.SolrCore [merito_shard1_replica2] Registered new searcher Searcher#5747d7f5[merito_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(6.5.1):C1)))}
10:24:50.967 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
10:24:50.970 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher#56f7a479[merito_shard1_replica2] main]
10:24:50.971 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.SolrCore [merito_shard1_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore#7d455130
10:24:50.972 INFO (searcherExecutor-584-thread-1-processing-n:192.168.94.133:8501_solr x:merito_shard1_replica2 s:shard1 c:merito r:core_node2) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher#56f7a479[merito_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(6.5.1):C1)))}
10:24:50.972 INFO (searcherExecutor-584-thread-1-processing-n:192.168.94.133:8501_solr x:merito_shard1_replica2 s:shard1 c:merito r:core_node2) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.QuerySenderListener QuerySenderListener done.
10:24:50.973 INFO (searcherExecutor-584-thread-1-processing-n:192.168.94.133:8501_solr x:merito_shard1_replica2 s:shard1 c:merito r:core_node2) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.c.SolrCore [merito_shard1_replica2] Registered new searcher Searcher#56f7a479[merito_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(6.5.1):C1)))}
10:24:50.980 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.merito.shard1.replica2
10:24:50.981 INFO (qtp575335780-18) [c:merito s:shard1 r:core_node2 x:merito_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=merito_shard1_replica2&qt=/admin/cores&action=RELOAD&wt=javabin&version=2} status=0 QTime=10388
10:24:51.175 INFO (qtp575335780-19) [ ] o.a.s.c.RequestParams conf resource params.json loaded . version : 20
10:24:51.176 INFO (qtp575335780-19) [ ] o.a.s.c.RequestParams request params refreshed to version 20
10:24:51.178 INFO (qtp575335780-19) [ ] o.a.s.c.SolrResourceLoader [merito_shard1_replica2] Added 51 libs to classloader, from paths: [/home/user/Desktop/Solr/instance2/contrib/clustering/lib, /home/user/Desktop/Solr/instance2/contrib/extraction/lib, /home/user/Desktop/Solr/instance2/contrib/langid/lib, /home/user/Desktop/Solr/instance2/contrib/velocity/lib, /home/user/Desktop/Solr/instance2/dist]
10:24:51.198 INFO (qtp575335780-19) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.1
10:24:51.220 INFO (qtp575335780-19) [ ] o.a.s.s.IndexSchema [merito_shard1_replica2] Schema name=example-basic-bdm
10:24:51.496 INFO (qtp575335780-19) [ ] o.a.s.s.IndexSchema Loaded schema example-basic-bdm/1.6 with uniqueid field id
After each block it starts againg with o.a.s.c.CoreContainer Reloading SolrCore .
Do you have any suggestion on how to prevent this behaviour? The logs are filling with this spam messages.
Thanks in advance

Answer 1

You are calling it from a cron or windows service. Check solr server interaction with other servers from resource io monitor and identify the process.
So i was in same situation as well my solr instance was reloading after every 2 mins. Turns out there was a cron(Windows service) running on the server that was continuously reloading it after 2 mins so as to add new synonyms from dictionary continuously.
I controlled that now.