aurora-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bill Farner (JIRA)" <>
Subject [jira] [Commented] (AURORA-1061) Scheduler cluster deadlock during leader election if zookeeper connectivity is interrupted
Date Fri, 19 Jun 2015 23:03:03 GMT


Bill Farner commented on AURORA-1061:

If you witness this again, please try to capture thread stack traces.  That will make it significantly
easier to find the source of the deadlock and explore fixes.

> Scheduler cluster deadlock during leader election if zookeeper connectivity is interrupted
> ------------------------------------------------------------------------------------------
>                 Key: AURORA-1061
>                 URL:
>             Project: Aurora
>          Issue Type: Bug
>          Components: Reliability, Scheduler
>            Reporter: Kevin Sweeney
> Observed in production: the scheduler cluster deadlocked with all nodes waiting for a
> This manifested as 6 singleton_candidate_ nodes in ZooKeeper. Since there were 5 schedulers
one was double-counted. Inspecting the contents, 2 nodes contained the same IP address.
> Restarting the scheduler with duplicated znodes allowed a new leader to take over, mitigating
the issue.
> This is a bug in the leader election library. The scheduler created a new ephemeral znode
after a connection loss with the zk ensemble without realizing that its session hadn't expired
and thus its previous znode was still valid. What's worse, that previous znode was the winner
of the election, making other schedulers sit idle as the leader waited for itself.
> Logs from the stuck leader confirm: it attempted to do a create, got a ConnectionLoss,
then reconnected with the same session id (0xf4af4a49d4ef61c), created a new node (singleton_candidate_0000003471).
Meanwhile the previous created ephemeral znode (singleton_candidate_0000003470) persisted
as the session was kept alive.
> {noformat}
> I0120 23:03:30.206 THREAD133 org.apache.zookeeper.ClientCnxn$ Client session
timed out, have not heard from server in 6670ms for sessionid 0xf4af4a49d4ef61c, closing socket
connection and attempting reconnect
> W0120 23:03:30.315 THREAD1 com.twitter.common.zookeeper.Group$2.get: Temporary error
trying to join group at path: /twitter/service/meso
> s/prod/scheduler
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
for /twitter/service/mesos/prod/scheduler/singleton_candidate_
>         at org.apache.zookeeper.KeeperException.create(
>         at org.apache.zookeeper.KeeperException.create(
>         at org.apache.zookeeper.ZooKeeper.create(
>         at com.twitter.common.zookeeper.Group$ActiveMembership.join(
>         at com.twitter.common.zookeeper.Group$2.get(
>         at com.twitter.common.zookeeper.Group$2.get(
>         at com.twitter.common.util.BackoffHelper.doUntilResult(
>         at com.twitter.common.zookeeper.Group.join(
>         at com.twitter.common.zookeeper.CandidateImpl.offerLeadership(
>         at com.twitter.common.zookeeper.SingletonService.lead(
>         at
>         at
>         at com.twitter.common.application.AppLauncher.launch(
>         at com.twitter.common.application.AppLauncher.launch(
>         at
> I0120 23:03:30.366 THREAD133 org.apache.zookeeper.ClientCnxn$SendThread.startConnect:
Opening socket connection to server
> I0120 23:03:30.366 THREAD133 org.apache.zookeeper.ClientCnxn$SendThread.primeConnection:
Socket connection established to, initiating
> I0120 23:03:30.368 THREAD133 org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult:
Session establishment complete on server, sessionid
= 0xf4af4a49d4ef61c, negotiated timeout = 10000
> I0120 23:03:31.683 THREAD1 com.twitter.common.zookeeper.Group$ActiveMembership.join:
Set group member ID to singleton_candidate_0000003471
> I0120 23:07:22.548 THREAD134 com.twitter.common.zookeeper.CandidateImpl$4.onGroupChange:
Candidate /twitter/service/mesos/prod/scheduler/singleton_candidate_0000003471 waiting for
the next leader election, current voting: [singleton_candidate_0000003470, singleton_candidate_0000003471,
singleton_candidate_0000003476, singleton_candidate_0000003474, singleton_candidate_0000003472,
> {noformat}

This message was sent by Atlassian JIRA

View raw message