hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3332) [Umbrella] Unified Resource Statistics Collection per node
Date Wed, 11 Mar 2015 04:19:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14356260#comment-14356260

Vinod Kumar Vavilapalli commented on YARN-3332:

Chose the service model because the machine level big picture is fragmented between YARN and
HDFS (and HBase etc) - having a lower level common statistics layer is useful.

I anyways needed a service to expose an API for both admins/users as well as external systems
beyond HDFS too - I can imagine tools being built on top of this.

That said, it doesn't need to be service or library. I can think of a library that wires into
the exposed API, though I haven't found uses for that yet.

> [Umbrella] Unified Resource Statistics Collection per node
> ----------------------------------------------------------
>                 Key: YARN-3332
>                 URL: https://issues.apache.org/jira/browse/YARN-3332
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: Design - UnifiedResourceStatisticsCollection.pdf
> Today in YARN, NodeManager collects statistics like per container resource usage and
overall physical resources available on the machine. Currently this is used internally in
YARN by the NodeManager for only a limited usage: automatically determining the capacity of
resources on node and enforcing memory usage to what is reserved per container.
> This proposal is to extend the existing architecture and collect statistics for usage
b​eyond​ the existing use­cases.
> Proposal attached in comments.

This message was sent by Atlassian JIRA

View raw message