Mailing-List: contact issues-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
Date: Fri, 17 Apr 2015 22:21:59 +0000 (UTC)
From: "Bill Farner (JIRA)" <jira@apache.org>
To: issues@aurora.apache.org
Message-ID: <JIRA.12772206.1423009870000.35903.1429309319299@Atlassian.JIRA>
In-Reply-To: <JIRA.12772206.1423009870000@Atlassian.JIRA>
References: <JIRA.12772206.1423009870000@Atlassian.JIRA>
 <JIRA.12772206.1423009870732@arcas>
Subject: [jira] [Comment Edited] (AURORA-1096) Scheduler updater should
 limit the number of job/instance events
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/AURORA-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304424#comment-14304424 ] 

Bill Farner edited comment on AURORA-1096 at 4/17/15 10:21 PM:
---------------------------------------------------------------

Clarification:

A cap on {{JobUpdateEvent}} s can only be applied dynamically.  When the limit is met, we should transition the update to the {{ERROR}} state.  We can't enforce this at update start time because it is effectively unbounded (a user can pause/resume forever).

The cap on {{JobInstanceUpdateEvent}} should be imposed at job update submission time, by calculating the maximum number of instance events permitted by the configuration.


was (Author: wfarner):
Clarification:

A cap on {{JobUpdateEvent}}s can only be applied dynamically.  When the limit is met, we should transition the update to the {{ERROR}} state.  We can't enforce this at update start time because it is effectively unbounded (a user can pause/resume forever).

The cap on {{JobInstanceUpdateEvent}} should be imposed at job update submission time, by calculating the maximum number of instance events permitted by the configuration.

> Scheduler updater should limit the number of job/instance events
> ----------------------------------------------------------------
>
>                 Key: AURORA-1096
>                 URL: https://issues.apache.org/jira/browse/AURORA-1096
>             Project: Aurora
>          Issue Type: Story
>          Components: Scheduler
>            Reporter: Maxim Khutornenko
>
> Large/flapping scheduler job updates may generate too many events in the update store. The update settings are fully controlled by the user and there is a potential for a misconfigured job update to completely overwhelm our in-memory DB storage with job update instance events. 
> For example, a large flapping update with {{max_per_shard_failures}} and {{max_total_failures}} set to max INT when left unattended can quickly consume all available RAM and kill the scheduler. A manual cleanup of the scheduler log would be needed to bring the scheduler up.
> This can be especially relevant with the introduction of update heartbeats  (AURORA-690) that can further exacerbate the problem (e.g. when {{blockIfNoPulseAfterMs}} set too low wrt the external service pulse rate).
> We need to cap the max per-job lifetime count of {{JobUpdateEvent}} and {{JobInstanceUpdateEvent}} instances. A nice bonus would be providing a hint in the UI when the event sequence is cut off.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)