aurora-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zameer Manji (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AURORA-1890) Job Update Pulse History is not durably stored
Date Mon, 13 Feb 2017 22:00:43 GMT

    [ https://issues.apache.org/jira/browse/AURORA-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864537#comment-15864537
] 

Zameer Manji commented on AURORA-1890:
--------------------------------------

The scheduler does the right thing on first pulse. However on failover, any coordinated updates
are immediately sent to BLOCKED_AWAITING_PULSE. This is because on scheduler startup pulse
state is reset to no pulse received. The code sets the timestamp to the last pulse received
to 0L:

{noformat}
    synchronized void initializePulseState(IJobUpdate update, JobUpdateStatus status) {
      pulseStates.put(update.getSummary().getKey(), new PulseState(
          status,
          update.getInstructions().getSettings().getBlockIfNoPulsesAfterMs(),
          0L));
    }
{noformat}

Would it be ok to set the timestamp to the first event after the most recent {{BLOCKED_AWAITING_PULSE}}?
We know for sure at that point in time that a pulse was received because of the state transition
from {{BLCOKED_AWAITING_PULSE}} to some other event.

Also could you describe "significant" write volume? I can imagine if the pulse interval was
in the seconds and there are thousands of updates perhaps it would be too much. However we
could prevent excessively small pulse intervals.

> Job Update Pulse History is not durably stored
> ----------------------------------------------
>
>                 Key: AURORA-1890
>                 URL: https://issues.apache.org/jira/browse/AURORA-1890
>             Project: Aurora
>          Issue Type: Bug
>            Reporter: Zameer Manji
>
> I have experienced the following problem with pulse updates. To reproduce:
> 1. Create an update with a pulse timeout of 1h
> 2. Send a pulse to get the update going.
> 3. Failover the scheduler immediately after.
> 4. Observe that the update is awaiting another pulse right after the failover.
> This is because the {{JobUpdateControllerImpl}} stores pulse history and state in memory
in {{PulseHandler}}. On scheduler startup, the pulse state is reset to no pulse received.
> We can solve this by durably storing the timestamp of the last pulse received in storage.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message