I get below exception at solrcloud client end during data ingestion:
ERROR com.aexp.ims.atworks.ingestion.service.impl.IngestionServiceImpl - Solr Exception
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://tsnet1:8888/solr/TSEACH_shard1_replica2, http://tsnet2:8888/solr/TSEACH_shard2_replica2, http://tsnet3:8888/solr/TSEACH_shard1_replica1, http://tsnet4:8888/solr/TSEACH_shard2_replica1]
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://tsnet5:8888/solr/TSEACH_shard2_replica2
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://tsnet2:8888 refused
Caused by: java.net.ConnectException: Connection refused
Solr log error:
ERROR update.UpdateLog - Error inspecting tlog tlog{file=/data/tsearch/solr/TSEACH_shard2_replica1/data/tlog/tlog.0000000000000005405 refcount=2}
ERROR update.UpdateLog - Error inspecting tlog tlog{file=/data/tsearch/solr/TSEACH_shard2_replica1/data/tlog/tlog.0000000000000005406 refcount=2}
ERROR update.UpdateLog - Error inspecting tlog tlog{file=/data/tsearch/solr/TSEACH_shard4_replica1/data/tlog/tlog.0000000000000005433 refcount=2}
ERROR update.UpdateLog - Error inspecting tlog tlog{file=/data/tsearch/solr/TSEACH_shard4_replica1/data/tlog/tlog.0000000000000005434 refcount=2}
Any help to resolve above error would be greatly appreciated?
I had a similar issue.
There could be many reasons to cause this exception. This is a one general error message masking the inner details.
While I cant offer a concrete solution to this problem, here is how I fixed my issue.
1. Check logs of all nodes.
Note that in cloud mode server logs are spread across nodes.
I was running cloud on single node. I could do this:
tail -100 example/cloud/node*/logs/solr.log
2. Find out inner cause and its fix
Do you see an exception?
Great.
Now google/lookup for the behind the scenes exception.
In my case, the group by query with group.ngroups=0 was the issue. It was fixed in 6.3 https://issues.apache.org/jira/browse/SOLR-4164
So, I upgraded solr and it solved
I had similar issue when my solr server was accidentally stopped.
I restarted it from bin/platform using following command and it worked:
ant startSolrServer
Related
I have stargate 1.0.38 running fine in my DEV server. I am able to use stargate rest api to get auth_token and running insert, select queries.
Yesterday, I have created an index Cql3SolrSecondaryIndex for a table in my Cassandra DSE 6.8. Then I see bellow error in stargate log. After that, I dropped that index. But even after dropping the index, i still see bellow error in stargate log. I also try to stop/start stargate but the still see same error.
ERROR [MigrationStage:1] 2021-10-15 00:47:13,593 PullRequestScheduler.java:245 - Configuration exception merging remote schema
org.apache.cassandra.exceptions.ConfigurationException: Unable to find custom indexer class 'com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex'
at org.apache.cassandra.utils.FBUtilities.classForName(FBUtilities.java:493)
at org.apache.cassandra.schema.IndexMetadata.getCustomIndexClass(IndexMetadata.java:190)
at org.apache.cassandra.schema.IndexMetadata.validate(IndexMetadata.java:131)
at org.apache.cassandra.schema.Indexes.lambda$validate$2(Indexes.java:168)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.cassandra.schema.Indexes.validate(Indexes.java:168)
at org.apache.cassandra.schema.TableMetadata.validate(TableMetadata.java:512)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.cassandra.schema.KeyspaceMetadata.validate(KeyspaceMetadata.java:112)
at org.apache.cassandra.schema.KeyspaceMetadata.<init>(KeyspaceMetadata.java:85)
at org.apache.cassandra.schema.KeyspaceMetadata.create(KeyspaceMetadata.java:167)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:1154)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspaces(SchemaKeyspace.java:1769)
at org.apache.cassandra.schema.SchemaManager.merge(SchemaManager.java:893)
at org.apache.cassandra.schema.SchemaManager.mergeAndAnnounceVersion(SchemaManager.java:877)
at org.apache.cassandra.schema.PullRequestScheduler.lambda$sendPullRequest$2(PullRequestScheduler.java:240)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:88)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
Caused by: java.lang.ClassNotFoundException: com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex not found by io.stargate.db.dse [1]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1597)
at org.apache.felix.framework.BundleWiringImpl.access$300(BundleWiringImpl.java:79)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1982)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.cassandra.utils.FBUtilities.classForName(FBUtilities.java:489)
... 24 common frames omitted
Because of this error, I am still able to get auth_token but all the select queries get this error
{
"description": "Resource not found: keyspace 'test' not found",
"code": 404
}
Please help me to fix this issue.
Stargate does not currently support advanced workloads like Search and Graph. I think you might have to drop and recreate that keyspace without the Solr index for it to work again since the schema still exists on the other nodes.
This issue has been documented here. There has also been a request made to support Solr here.
#David, I was able to drop the DSE Search index, perform a rolling restart of my DSE node(s) and then Stargate node(s) and it started up just fine without any errors. I ensure all my prior data existed just fine and was able to validate running basic CRUD operations using Stargate REST, GraphQL & Document APIs without any issues post that.
I have been indexing for quite a long time but now I am not able to do so. I keep on getting the following error.
INFO [Thread-80] (00002SB6) [SolrIndexerJob] Started indexer cronjob.
ERROR [Thread-80] (00002SB6) [Job] Caught throwable de/hybris/platform/solrfacetsearch/config/IndexConfig
java.lang.NoClassDefFoundError: de/hybris/platform/solrfacetsearch/config/IndexConfig
at ma.glasnost.orika.generated.Orika_FacetSearchConfig_FacetSearchConfig_Mapper45623933135018$4.mapAtoB(Orika_FacetSearchConfig_FacetSearchConfig_Mapper45623
933135018$4.java)
at ma.glasnost.orika.impl.mapping.strategy.UseCustomMapperStrategy.map(UseCustomMapperStrategy.java:67)
at ma.glasnost.orika.impl.MapperFacadeImpl.map(MapperFacadeImpl.java:735)
at ma.glasnost.orika.impl.MapperFacadeImpl.map(MapperFacadeImpl.java:714)
at ma.glasnost.orika.impl.ConfigurableMapper.map(ConfigurableMapper.java:150)
at de.hybris.platform.solrfacetsearch.config.impl.DefaultFacetSearchConfigService.getConfiguration(DefaultFacetSearchConfigService.java:51)
at de.hybris.platform.solrfacetsearch.indexer.cron.AbstractIndexerJob.getFacetSearchConfig(AbstractIndexerJob.java:70)
at de.hybris.platform.solrfacetsearch.indexer.cron.SolrIndexerJob.performIndexingJob(SolrIndexerJob.java:49)
at de.hybris.platform.multicountry.solr.indexer.cron.impl.MulticountrySolrIndexerJob.performIndexingJob(MulticountrySolrIndexerJob.java:73)
at de.hybris.platform.solrfacetsearch.indexer.cron.AbstractIndexerJob.perform(AbstractIndexerJob.java:40)
at de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob.performCronJob(ServicelayerJob.java:38)
at de.hybris.platform.cronjob.jalo.Job.execute(Job.java:1390)
at de.hybris.platform.cronjob.jalo.Job.performImpl(Job.java:814)
at de.hybris.platform.cronjob.jalo.Job.performImpl(Job.java:732)
at de.hybris.platform.cronjob.jalo.Job.perform(Job.java:644)
at de.hybris.platform.servicelayer.cronjob.impl.DefaultCronJobService.performCronJob(DefaultCronJobService.java:86)
at de.hybris.platform.solrfacetsearchbackoffice.wizards.BaseSolrIndexerWizardStep$WizardCronJobAsyncOperation.execute(BaseSolrIndexerWizardStep.java:158)
at com.hybris.cockpitng.engine.impl.DefaultWidgetInstanceManager$1.getResult(DefaultWidgetInstanceManager.java:206)
at com.hybris.cockpitng.engine.operations.ResultLongOperation.execute(ResultLongOperation.java:52)
at com.hybris.cockpitng.engine.operations.LongOperation.run(LongOperation.java:205)
at java.lang.Thread.run(Thread.java:748)
Not sure which change is causing this issue.
Tried to setup new fresh hybris suite still the same issue. Followed hybris answers which mentioned to perform ant clean all and server startup, didn't work. Created new index and then performed indexing still the same. I am able to open solr admin but not able to index anything.
Any help would be really appreciated.
Just restarted the server. It worked for me!
After moving from Solr 4.4 to Solr 4.6.1, I am getting the below Exception while updating my Indexes using the Data Import Handler. Does anybody have any ideas on why this is happening?
ERROR - 2014-02-18 09:39:35.232; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Bad Request
request: http://10.200.131.174:8080/solr/collection1/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2F10.200.131.173%3A8080%2Fsolr%2Fcollection1%2F&wt=javabin&version=2
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
ERROR - 2014-02-18 09:39:35.244; org.apache.solr.update.StreamingSolrServers$1; error
org.apache.solr.common.SolrException: Bad Request
This is a known issue for Solr versions greater than 4.5. I too ran into this while running SolrCloud with 2 shards. It seems that the update works fine on the server running DIH, but fails when it is forwarded to another shard or replica.
Here's the related jira issue. And here's another unlucky soul like you and I who also faced this error.
As far as I know, the only workaround currently is to go back to a Solr version 4.5 or below...
My slave server is replicating with master, that has very large index. however, I am seeing this error :
SnapPull failed :org.apache.solr.common.SolrException: Unable to download _avx.fdx completely. Downloaded 0!=3748828, very frequently
Unfortunately, the stacktrace is very thin and the error message is not very descriptive either, so I left to debug this issue myself, i did following :
Searched on stackoverflow to see if anybody else facing the issue and found this posts :- solr ReplicationHandler - SnapPull failed to download files, wherein the user had to replace the http jars because of some garbage in the response. Though I din't see such message in the response, I went and replaced the http common jars but not much use!.
Have even tried the lucene mailinglist, but haven't got affirmative answer.
Additionally, I also see many index.timestamp directories, which I believe solr creates to download the index files during replication. Curious why aren't they getting deleted ?
If somebody has any faced this issue, please throw some light, would be very grateful.
I have solrj client with infinite timeout(Solr4)
server.server.setSoTimeout(0)
server.server.setConnectionTimeout(0)
When I index my data I have many timeouts on server side.
Where can I update server side timeouts in solrconfig.xml or possible tomcat config?
Client side exception:
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
Server side exception:
Jan 31, 2013 8:55:54 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Read timed out
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:159)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:751)
We had the same problem with Solr 4. We solved this after reading a blog post by Uwe Schindler (a Solr commiter).
With Solr 4 and several Solr 3 versions, you have to let an important share of your RAM free so that the system can use properly the mmap system call. This can be subtle depending on your system configuration (the blog post gives a plenty of informations on that point). In our case, this solved the problem: we could finally index without any more timeout issue.
the info for tomcat server.xml config will solve this. we got same stack trace and the below solved it for us:
http://forums.alfresco.com/ja/node/8458