hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Lu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5638) Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery
Date Tue, 20 Sep 2016 21:25:20 GMT

     [ https://issues.apache.org/jira/browse/YARN-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Li Lu updated YARN-5638:
    Attachment: YARN-5638-trunk.v2.patch

Thanks for the review [~rohithsharma]! I addressed most of you comments except those two:

bq. Can happensBefore comparison method name can be changed something meaningful ? May we
can define comparator method itself.
Happens-before has very concrete meaning in distributed system theory, as defined in Lamport's
"Time, Clocks, and the Ordering of Events in a Distributed System" (http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf)
Here, we're assigning each collector data a timestamp, and then we use timestamps to reason
about happens before order in the system. Personally I'd prefer using this formal definition
to capture our use case here. 

bq. In stamped method, I think check for both rmIdentifiers && version?
Did not quite get the point here... I think we're checking both fields? 

About the design question, a pull (IIUC, pulling collector data from the RM) based method
works. The current approach offloads the burden of deciding where collectors should run from
the RM. At the early stage we can also reuse some well established mechanisms with heartbeat.
I believe most them are for engineering reasons, though. 

One thing to note is that we will not send known collector information from NMs to the RM
in *every* heartbeat. A collector's information got sent only when it needs registration to
the RM, or the RM needs resync. On the other hand, the RM will only send related collector's
information to each of the NMs. 

> Introduce a collector timestamp to uniquely identify collectors creation order in collector
> -----------------------------------------------------------------------------------------------------
>                 Key: YARN-5638
>                 URL: https://issues.apache.org/jira/browse/YARN-5638
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-5638-trunk.v1.patch, YARN-5638-trunk.v2.patch
> As discussed in YARN-3359, we need to further identify timeline collectors' creation
order to rebuild collector discovery data in the RM. This JIRA proposes to use <rm_timestamp,
logical_version_number> to order collectors for each application in the RM. This timestamp
can then be used when a standby RM becomes active and rebuild collector discovery data. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message