lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Christiaens <mark.g.j.christia...@gmail.com>
Subject Solr 4.10.3 goes into recovery for unknown reason
Date Fri, 03 Jun 2016 10:10:40 GMT
We have a SolrCloud setup (within a Cloudera deployment) using Solr
4.10.3.  The Solr cluster consists of 2 nodes.  Both their backing store is
on HDFS (not on the local file system).  Once every 1-2 weeks, the system
goes into recovery without apparent reason.  Digging through the logs it
looks something like this:

>From time to time, when the load is high, the leader/replica election goes
wrong. We see that

May 31, 10:31:46.524 AM ERROR org.apache.solr.core.SolrCore
org.apache.solr.common.SolrException: ClusterState says we are the leader (
http://grbbd1nodp06.core.local:8983/solr/lily_entity_CUSTOMER_shard1_replica2),
but locally we don't think so. Request came from null

When that happens, the cluster starts recovering:

May 31, 10:33:04.621 AM INFO org.apache.solr.cloud.RecoveryStrategy
Publishing state of core lily_entity_CUSTOMER_shard2_replica2 as
recovering, leader is
http://grbbd1nodp05.core.local:8983/solr/lily_entity_CUSTOMER_shard2_replica1/
and I am
http://grbbd1nodp06.core.local:8983/solr/lily_entity_CUSTOMER_shard2_replica2/

Apparently, that doesn't go too smoothly either. First it tries something
called "PeerSync" that fails and then Solr goes to "replication". I suspect
that "PeerSync" is a recovery strategy where the last N updates are
transferred from one node to other in order for that other node to catch
up. And "replication" probably is copying the entire contents of a node to
another.

PeerSync Recovery was not successful - trying replication.
core=lily_models_shard2_replica1
grbbd1nodp06.core.local INFO May 31, 2016 10:32 AM RecoveryStrategy
Starting Replication Recovery. core=lily_models_shard2_replica1

Apparently the recovery of a Solr node happens piecemeal. For some pieces,
the replication or PeerSync seems to fail, for others it does not fail.

Wait 2.0 seconds before trying to recover again (1)
grbbd1nodp06.core.local ERROR May 31, 2016 10:32 AM RecoveryStrategy
Error while trying to recover:org.apache.solr.common.SolrException:
Replication for recovery failed.
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:168)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:448)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:237)
grbbd1nodp06.core.local ERROR May 31, 2016 10:32 AM RecoveryStrategy
Recovery failed - trying again... (0)
core=lily_entity_CUSTOMER_shard2_replica2
grbbd1nodp06.core.local ERROR May 31, 2016 10:32 AM ReplicationHandler
SnapPull failed :org.apache.solr.common.SolrException: Index fetch failed :
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:573)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:310)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:349)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:165)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:448)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:237)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://grbbd1clup01-ns/solr/lily_entity_CUSTOMER/core_node2/data/index.20160429093321915/segments_8hp9
at
org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1218)
at
org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1210)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1210)
at
org.apache.solr.store.hdfs.HdfsDirectory$HdfsIndexInput.<init>(HdfsDirectory.java:205)
at
org.apache.solr.store.hdfs.HdfsDirectory.openInput(HdfsDirectory.java:136)
at
org.apache.solr.store.blockcache.BlockDirectory.openInput(BlockDirectory.java:124)
at
org.apache.solr.store.blockcache.BlockDirectory.openInput(BlockDirectory.java:144)
at
org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:198)
at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:341)
at org.apache.solr.handler.SnapPuller.hasUnusedFiles(SnapPuller.java:623)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:456)
... 5 more

After 10 minutes or so, the system does seem to recover.

I dug through the Solr bugs but could not find this exact issue.  I'm
particularly intrigued by the reason why the leader/replica election seems
to go wrong.  Any suggestions?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message