aurora-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Erb (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AURORA-1944) Aurora is unable to elect leader after losing ZK for an extended period of time
Date Sun, 06 Aug 2017 21:12:00 GMT

    [ https://issues.apache.org/jira/browse/AURORA-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115938#comment-16115938
] 

Stephan Erb edited comment on AURORA-1944 at 8/6/17 9:11 PM:
-------------------------------------------------------------

On a first glance, there appear to be two problems and we should probably fix both of them:

* When ZK is down at startup, {{CuratorServiceGroupMonitor.start}} fails with {{Failed to
begin monitoring service group.}}. The start is not re-tried in any form but instead a shutdown
is triggered.
* The scheduler shutdown fails with {{State transition from CONSTRUCTED to STOPPED is not
allowed.}}

With this patch applied, unit tests, e2e tests, and scheduler start with ZK down seem all
to work on first sight in the vagrant environment:
{code}
diff --git a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java
b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java
index eba56be..4551b44 100644
--- a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java
+++ b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java
@@ -69,10 +69,7 @@ class CuratorServiceGroupMonitor implements ServiceGroupMonitor {
   @Override
   public void start() throws MonitorException {
     try {
-      // NB: This blocks on an initial group population to emulate legacy ServerSetMonitor
behavior;
-      // asynchronous population is an option using NORMAL or POST_INITIALIZED_EVENT StartModes
-      // though.
-      groupCache.start(PathChildrenCache.StartMode.BUILD_INITIAL_CACHE);
+      groupCache.start(PathChildrenCache.StartMode.NORMAL);
     } catch (Exception e) {
       throw new MonitorException("Failed to begin monitoring service group.", e);
     }
{code}


[~jsirois] what do you think?



was (Author: stephanerb):
On a first glance, there appear to be two problems and we should probably fix both of them:

* When ZK is down at startup, {{CuratorServiceGroupMonitor.start}} fails with {{Failed to
begin monitoring service group.}}. The start is not re-tried in any form but instead a shutdown
is triggered.
* The scheduler shutdown fails with {{State transition from CONSTRUCTED to STOPPED is not
allowed.}}

With this patch applied, unit tests, e2e tests, and scheduler start with ZK down seem all
to work on first sight:
{code}
diff --git a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java
b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java
index eba56be..4551b44 100644
--- a/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java
+++ b/src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceGroupMonitor.java
@@ -69,10 +69,7 @@ class CuratorServiceGroupMonitor implements ServiceGroupMonitor {
   @Override
   public void start() throws MonitorException {
     try {
-      // NB: This blocks on an initial group population to emulate legacy ServerSetMonitor
behavior;
-      // asynchronous population is an option using NORMAL or POST_INITIALIZED_EVENT StartModes
-      // though.
-      groupCache.start(PathChildrenCache.StartMode.BUILD_INITIAL_CACHE);
+      groupCache.start(PathChildrenCache.StartMode.NORMAL);
     } catch (Exception e) {
       throw new MonitorException("Failed to begin monitoring service group.", e);
     }
{code}


[~jsirois] what do you think?


> Aurora is unable to elect leader after losing ZK for an extended period of time
> -------------------------------------------------------------------------------
>
>                 Key: AURORA-1944
>                 URL: https://issues.apache.org/jira/browse/AURORA-1944
>             Project: Aurora
>          Issue Type: Bug
>          Components: Scheduler
>         Environment: Running on 0.17.0
>            Reporter: Renan DelValle
>         Attachments: aurora-0.log, aurora-1.log, aurora-2.log
>
>
> Using Apache Curator as the Zookeeper library causes an issue where Aurora is unable
to elect a leader if Zookeeper loses quorum for an extended period of time.
> Scheduler seems to crash around:
> {{W0802 14:01:14.436 [TaskEventBatchWorker, SchedulerLifecycle] Failed to leave leadership:
org.apache.aurora.common.zookeeper.SingletonService$LeaveException: Failed to abdicate leadership
of group at /aurora/scheduler}}
> When the init system brings the scheduler back up, it is unable to elect a leader if
ZK is still down.
> Specifically, the redirect monitor fails:
> {{E0802 14:09:37.063 [RedirectMonitor STARTING, GuavaUtils$LifecycleShutdownListener]
Service: RedirectMonitor [FAILED] failed unexpectedly. Triggering shutdown.}}
> Leading to every scheduler showing the following:
> {{W0802 14:16:34.646 [qtp576711849-43, LeaderRedirect] No serviceGroupMonitor in host
set, will not redirect despite not being leader.}}
> Once the scheduler enters this state, it is unable to snap out of it until it is manually
restarted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message