hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5638) Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery
Date Thu, 15 Sep 2016 22:20:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494688#comment-15494688

Li Lu commented on YARN-5638:

Updated the description of this JIRA. What we need here is not a new type of "collector id",
but to store timestamp data in the RMs and NMs for the collectors. This can address the problem
when we rebuild collector status for a new active rm, as discussed in YARN-3359: 

bq. when one application has two different attempts running (due to some network problems,
for example) and the RM is trying to rebuild collector status, the RM needs to know which
collector is for the latest app attempt and which one is for the stale attempt.

We do not necessarily need to associate collectors to application attempts. Actually, according
to timeline server v2 design, we should only associate app collectors to applications. However,
when maintaining collector data in RMs and NMs, we can store the timestamp of each collector.
In this way, when the RM needs to rebuild collector status, it can gather all known collector
data from NMs, use the timestamp to decide the most recent state of the collectors, and then
rebuild all states. 

> Introduce a collector timestamp to uniquely identify collectors creation order in collector
> -----------------------------------------------------------------------------------------------------
>                 Key: YARN-5638
>                 URL: https://issues.apache.org/jira/browse/YARN-5638
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
> As discussed in YARN-3359, we need to further identify timeline collectors' creation
order to rebuild collector discovery data in the RM. This JIRA proposes to use <rm_timestamp,
logical_version_number> to order collectors for each application in the RM. This timestamp
can then be used when a standby RM becomes active and rebuild collector discovery data. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message