Mailing-List: contact issues-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
Date: Fri, 4 Aug 2017 22:00:04 +0000 (UTC)
From: "Renan DelValle (JIRA)" <jira@apache.org>
To: issues@aurora.apache.org
Message-ID: <JIRA.13092571.1501874139000.97653.1501884004659@Atlassian.JIRA>
In-Reply-To: <JIRA.13092571.1501874139000@Atlassian.JIRA>
References: <JIRA.13092571.1501874139000@Atlassian.JIRA> <JIRA.13092571.1501874139055@jira-lw-us.apache.org>
Subject: [jira] [Updated] (AURORA-1944) Aurora is unable to elect leader
 after losing ZK for an extended period of time
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 04 Aug 2017 22:00:10 -0000


     [ https://issues.apache.org/jira/browse/AURORA-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Renan DelValle updated AURORA-1944:
-----------------------------------
    Description: 
Using Apache Curator as the Zookeeper library causes an issue where Aurora is unable to elect a leader if Zookeeper loses quorum for an extended period of time.

Scheduler seems to crash around:

{{W0802 14:01:14.436 [TaskEventBatchWorker, SchedulerLifecycle] Failed to leave leadership: org.apache.aurora.common.zookeeper.SingletonService$LeaveException: Failed to abdicate leadership of group at /aurora/scheduler}}

When the init system brings the scheduler back up, it is unable to elect a leader if ZK is still down.

Specifically, the redirect monitor fails:

{{E0802 14:09:37.063 [RedirectMonitor STARTING, GuavaUtils$LifecycleShutdownListener] Service: RedirectMonitor [FAILED] failed unexpectedly. Triggering shutdown.}}

Leading to every scheduler showing the following:

{{W0802 14:16:34.646 [qtp576711849-43, LeaderRedirect] No serviceGroupMonitor in host set, will not redirect despite not being leader.}}

Once the scheduler enters this state, it is unable to snap out of it until it is manually restarted.


  was:
Using Apache Curator as the Zookeeper library causes an issue where Aurora is unable to elect a leader if Zookeeper loses quorum for an extended period of time.

Scheduler seems to crash around:
{{W0802 14:01:14.436 [TaskEventBatchWorker, SchedulerLifecycle] Failed to leave leadership: org.apache.aurora.common.zookeeper.SingletonService$LeaveException: Failed to abdicate leadership of group at /aurora/scheduler }}

When the init system brings the scheduler back up, it is unable to elect a leader if ZK is still down.

Once the scheduler enters this state, it is unable to snap out of it until it is manually restarted.


> Aurora is unable to elect leader after losing ZK for an extended period of time
> -------------------------------------------------------------------------------
>
>                 Key: AURORA-1944
>                 URL: https://issues.apache.org/jira/browse/AURORA-1944
>             Project: Aurora
>          Issue Type: Bug
>          Components: Scheduler
>         Environment: Running on 0.17.0
>            Reporter: Renan DelValle
>         Attachments: aurora-0.log, aurora-1.log, aurora-2.log
>
>
> Using Apache Curator as the Zookeeper library causes an issue where Aurora is unable to elect a leader if Zookeeper loses quorum for an extended period of time.
> Scheduler seems to crash around:
> {{W0802 14:01:14.436 [TaskEventBatchWorker, SchedulerLifecycle] Failed to leave leadership: org.apache.aurora.common.zookeeper.SingletonService$LeaveException: Failed to abdicate leadership of group at /aurora/scheduler}}
> When the init system brings the scheduler back up, it is unable to elect a leader if ZK is still down.
> Specifically, the redirect monitor fails:
> {{E0802 14:09:37.063 [RedirectMonitor STARTING, GuavaUtils$LifecycleShutdownListener] Service: RedirectMonitor [FAILED] failed unexpectedly. Triggering shutdown.}}
> Leading to every scheduler showing the following:
> {{W0802 14:16:34.646 [qtp576711849-43, LeaderRedirect] No serviceGroupMonitor in host set, will not redirect despite not being leader.}}
> Once the scheduler enters this state, it is unable to snap out of it until it is manually restarted.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)