hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3815) [Aggregation] Application/Flow/User/Queue Level Aggregations
Date Fri, 03 Jul 2015 03:34:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612798#comment-14612798

Sangjin Lee commented on YARN-3815:

We don't have to make it at container level I think but also not necessary for AM to retain
and aggregate these values. AM could help to forward the values to per app timeline collector
but don't have to aggregate them. Vinod got more ideas on this in offline discussion. [~vinodkv],
can you comment on this?

Interesting. Could you or [~vinodkv] shed light on the idea? It would still need to be captured
in an entity or entities, right? I would think sending it as part of the container entities
would be simpler and more consistent (in that the per-app collector can simply look at all
container metrics as subject to aggregation). I'd love to hear more about this.

I think "per-container averages" is not equal to per-container resource usage. Understanding
application's real resource consumption/usage is one of the core use cases for new timeline
service at the beginning so I don't think we should rule out anything important here.

How is the per-container resource usage different than the per-container average described
in the summary? Could you kindly provide its definition?

No doubt understanding applications' real resource consumption/usage is critical. Between
the individual container resource usage (which are all captured), the aggregated resource
usage at the app/flow level (which the basic real time aggregation addresses), and the running
averages/max of the aggregated resource usage at the app/flow level, I think it definitely
covers that need. What would be the gap that's not addressed by the above data?

> [Aggregation] Application/Flow/User/Queue Level Aggregations
> ------------------------------------------------------------
>                 Key: YARN-3815
>                 URL: https://issues.apache.org/jira/browse/YARN-3815
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Junping Du
>            Assignee: Junping Du
>            Priority: Critical
>         Attachments: Timeline Service Nextgen Flow, User, Queue Level Aggregations (v1).pdf,
aggregation-design-discussion.pdf, hbase-schema-proposal-for-aggregation.pdf
> Per previous discussions in some design documents for YARN-2928, the basic scenario is
the query for stats can happen on:
> - Application level, expect return: an application with aggregated stats
> - Flow level, expect return: aggregated stats for a flow_run, flow_version and flow 
> - User level, expect return: aggregated stats for applications submitted by user
> - Queue level, expect return: aggregated stats for applications within the Queue
> Application states is the basic building block for all other level aggregations. We can
provide Flow/User/Queue level aggregated statistics info based on application states (a dedicated
table for application states is needed which is missing from previous design documents like
HBase/Phoenix schema design). 

This message was sent by Atlassian JIRA

View raw message