lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manuel Le Normand <>
Subject Wrong leader election leads to shard removal
Date Wed, 14 Aug 2013 08:00:44 GMT
My solr cluster runs on RH Linux with tomcat7 servlet.
NumOfShards=40, replicationFactor=2, 40 servers each has 2 replicas. Solr

For experimental reasons I splitted my cluster to 2 sub-clusters, each
containing a single replica of each shard.
When connecting back these sub-clusters the sync failed (more than 100 docs
indexed per shard) so a replication process started on sub-cluster #2. Due
to transient storage limitations needed for the replication process, I
removed all the index from sub-cluster #2 before connecting it back, then I
connected sub-cluster #2's servers in 3-4 bulks to avoid high disk load.
The first bulk replications worked well, but after a while an internal
script pkilled all the solr instances, some while replicating. After
starting back the servlet I discovered the disaster - on part of the
replicas that were in a replicating stage there was a wrong zookeeper
leader election - good state replicas (sub-cluster 1) replicated from empty
replicas (sub-cluster 2) ending up in removing all documents in these

These are the logs from solr-prod32 (sub cluster #2 - bad state) - the
shard1_replica1 is elected to be leader although it was not before the
replication process (and shouldn't have the higher version number):

2013-08-13 13:39:15.838 [INFO ] Enough replicas found to
2013-08-13 13:39:15.838 [INFO ] I may be the new leader -
try and sync
2013-08-13 13:39:15.839 [INFO ] Sync
replicas to http://solr-prod32:5050/solr/raw shard1_replica1/
 2013-08-13 13:39:15.841 [INFO ]
org.apache.solr.client.solrj.impl.HttpClientUtil Creating new http client,
2013-08-13 13:39:15.844 [INFO ] org.apache.solr.update.PeerSync PeerSync:
core=raw_shard1_replica1 url=http://solr-prod32:8080/solr START replicas=[
http://solr-prod02:5080/solr/raw shard1_replica2/] nUpdates=100
2013-08-13 13:39:15.847 [INFO I org.apache.solr.update.PeerSync PeerSync:
core=raw shard1_replica1 url=http://solr-prod32:8080/solr DONE. We have
no versions. sync failed.
2013-08-13 13:39:15.847 [INFO ] Leader's
attempt to sync with shard failed, moving to the next canidate
2013-08-13 13:39:15.847 [INFO ] We failed sync, but we
have no versions - we can't sync in that case - we were active before, so
become leader anyway
2013-08-13 13:39:15.847 [INFO ] I am the new leader:
2013-08-13 13:39:15.847 [INFO ]
makePath: /collections/raw/leaders/shardl
2013-08-13 13:39:17.423 [INFO ]
A cluster state change: WatchedEvent state:SyncConnected
type:NodeDataChanged path:/clusterstate.json, has occurred - updating...
(live nodes size: 40)

While in solr-prod02 (sub cluster #1 - good state) I get:
2013-08-13 13:39:15.671 [INFO ]
publishing core=raw_shard1_replica2 state=down
2013-08-13 13:39:15.671 [INFO ]
numShards not found on descriptor - reading it from system property
2013-08-13 13:39:15.673 [INFO ] org.apache.solr.core.CoreContainer
registering core: raw_shard1_replica2
2013-08-13 13:39:15.673 [INFO ] Register
replica - core:raw_shard1_replica2 address:
http://so1r-prod02:8080/solrcollection:raw shard:shard1
2013-08-13 13:39:17.423 [INFO ]
A cluster state change: WatchedEvent stare:SyncConnected
type:NodeDataChanged path:/clusterstate.json, has occurred - updating...
(live nodes size: 40)
2013-08-13 13:39:17.480 [INFO ] We are
httpL//solr-prod02:8080/solr/raw_shard1_replica2/ and leader is
2013-08-13 13:39:17.481 [INFO ] No
LogReplay needed for core=raw_shard1_replica2
2013-08-13 13:39:17.481 [INFO ] Core
needs to recover:raw shard1_replica2
2013-08-13 13:39:17.481 [INFO ] org.apache.solr.update.DefaultSolrCoreState
Running recovery - first canceling any ongoing recovery
2013-08-13 13:39:17.485 [INFO
Updating cloud state from ZooKeeper...
2013-08-13 13:39:17.485 [INFO ]
Starting recovery process. core=raw_shard1_rep1ica2

Why was the leader elected wrongly??


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message