hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Botong Huang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-8696) FederationInterceptor upgrade: home sub-cluster heartbeat async
Date Wed, 22 Aug 2018 18:12:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Botong Huang updated YARN-8696:
    Attachment: YARN-8696.v2.patch

> FederationInterceptor upgrade: home sub-cluster heartbeat async
> ---------------------------------------------------------------
>                 Key: YARN-8696
>                 URL: https://issues.apache.org/jira/browse/YARN-8696
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Botong Huang
>            Assignee: Botong Huang
>            Priority: Major
>         Attachments: YARN-8696.v1.patch, YARN-8696.v2.patch
> Today in _FederationInterceptor_, the heartbeat to home sub-cluster is synchronous. After
the heartbeat is sent out to home sub-cluster, it waits for the home response to come back
before merging and returning the (merged) heartbeat result to back AM. If home sub-cluster
is suffering from connection issues, or down during an YarnRM master-slave switch, all heartbeat
threads in _FederationInterceptor_ will be blocked waiting for home response. As a result,
the successful UAM heartbeats from secondary sub-clusters will not be returned to AM at all.
Additionally, because of the fact that we kept the same heartbeat responseId between AM and
home RM, lots of tricky handling are needed regarding the responseId resync when it comes
to _FederationInterceptor_ (part of AMRMProxy, NM) work preserving restart (YARN-6127, YARN-1336),
home RM master-slave switch etc. 
> In this patch, we change the heartbeat to home sub-cluster to asynchronous, same as the
way we handle UAM heartbeats in secondaries. So that any sub-cluster down or connection issues
won't impact AM getting responses from other sub-clusters. The responseId is also managed
separately for home sub-cluster and AM, and they increment independently. The resync logic
becomes much cleaner. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message