Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@reef.apache.org
Date: Tue, 9 Aug 2016 17:13:20 +0000 (UTC)
From: "Andrey (JIRA)" <jira@apache.org>
To: dev@reef.apache.org
Message-ID: <JIRA.12996002.1470762788000.263248.1470762800460@Atlassian.JIRA>
In-Reply-To: <JIRA.12996002.1470762788000@Atlassian.JIRA>
References: <JIRA.12996002.1470762788000@Atlassian.JIRA> <JIRA.12996002.1470762788532@arcas>
Subject: [jira] [Created] (REEF-1511) timeout for Task Shutdown during IMRU
 recovery
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 09 Aug 2016 17:13:22 -0000

Andrey created REEF-1511:
----------------------------

             Summary: timeout for Task Shutdown during IMRU recovery
                 Key: REEF-1511
                 URL: https://issues.apache.org/jira/browse/REEF-1511
             Project: REEF
          Issue Type: Improvement
          Components: IMRU
            Reporter: Andrey


This related to fault tolerance implementation in PR-1251.
Currently recovery logic in IMRU driver is to wait for all task to move to a final state (failed or completed) before restarting the job check AreAllTasksInFinalState() in TryRecovery() method)
We've seen driver hanging for a long time waiting for few last tasks finalize.
Aborting tasks should be quick, so there is bug there, but we also can add logic in driver not to wait for all tasks to complete.
For instance: if 5% of tasks did not report final state withing expected period, release corresponding evaluators  and proceed with new job retry.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)