hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6694) Make AM more resilient to potential lost of any completed container notification
Date Wed, 11 May 2016 00:55:13 GMT
Ming Ma created MAPREDUCE-6694:
----------------------------------

             Summary: Make AM more resilient to potential lost of any completed container
notification
                 Key: MAPREDUCE-6694
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6694
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
            Reporter: Ming Ma


YARN tries to guarantee any completed container notification is delivered to AM under any
circumstance, YARN-1372 is an example to make sure for the case of RM restart. However, under
some corner cases, it is still possible a completed container notifications is lost or significantly
delayed. For example, if NM host becomes dead when RM fails over.

AM won't preempt reducers if it thought there is at least one mapper running.
{noformat}
  void preemptReducesIfNeeded() {
...
    if (assignedRequests.maps.size() > 0) {
      // there are assigned mappers
      return;
    }
...
{noformat}

Instead of completely depending on notification from RM, it can use TaskUmbilicalProtocol
to help to decide if there is any mapper running. That will make AM more resilient to any
bugs in YARN.
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message