hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3332) [Umbrella] Unified Resource Statistics Collection per node
Date Wed, 11 Mar 2015 05:08:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14356309#comment-14356309
] 

Zhijie Shen commented on YARN-3332:
-----------------------------------

It sounds a great proposal, thanks Vinod! I quick thought about the publishing channel of
the collected statistics. I'm not sure how different the access pattern would be, but just
thinking it out loudly, is it possible reuse the timeline service to distribute the node statistics,
getting rid of maintaining different but similar interfaces (or multiple data flow channels).
On step further, we can make the timeline service the main bus to transmit metrics from A
to B.

> [Umbrella] Unified Resource Statistics Collection per node
> ----------------------------------------------------------
>
>                 Key: YARN-3332
>                 URL: https://issues.apache.org/jira/browse/YARN-3332
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: Design - UnifiedResourceStatisticsCollection.pdf
>
>
> Today in YARN, NodeManager collects statistics like per container resource usage and
overall physical resources available on the machine. Currently this is used internally in
YARN by the NodeManager for only a limited usage: automatically determining the capacity of
resources on node and enforcing memory usage to what is reserved per container.
> This proposal is to extend the existing architecture and collect statistics for usage
b​eyond​ the existing use­cases.
> Proposal attached in comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message