hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3815) [Aggregation] Application/Flow/User/Queue Level Aggregations
Date Mon, 22 Jun 2015 16:08:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596129#comment-14596129

Junping Du commented on YARN-3815:

Thanks [~sjlee0] and [~jrottinghuis] for review and good comments in detail. [~jrottinghuis]'s
comments are pretty long and I could only reply part of it and will finish the left parts
tomorrow. :)

bq. For framework-specific metrics, I would say this falls on the individual frameworks. The
framework AM usually already aggregates them in memory (consider MR job counters for example).
So for them it is straightforward to write them out directly onto the YARN app entities. Furthermore,
it is problematic to add them to the sub-app YARN entities and ask YARN to aggregate them
to the application. Framework’s sub-app entities may not even align with YARN’s sub-app
entities. For example, in case of MR, there is a reasonable one-to-one mapping between a mapper/reducer
task attempt and a container, but for other applications that may not be true. Forcing all
frameworks to hang values at containers may not be practical. I think it’s far easier for
frameworks to write aggregated values to the YARN app entities.
AM currently leverage YARN's AppTimelineCollector to forward entities to backend storage,
so making AM talk directly to backend storage is not considered to be safe. It is also not
necessary too because the real difficulty here is to aggregate framework specific metrics
in other levels (flow, user and queue), because that beyond the life cycle of framework so
YARN have to take care of it. Instead of asking frameworks to handle specific metrics themselves,
I would like to propose to treat these metrics as "anonymous", it would pass both metrics
name and value to YARN's collector and YARN's collector could aggregate it and store as dynamic
column (under framework_specific_metrics column family) into app states table. So other (flow,
user, etc.) level aggregation on freamework metrics could happen based on this.

bq. app-to-flow online aggregation. This is more or less live aggregated metrics at the flow
level. This will still be based on the native HBase schema.
About flow online aggregation, I am not quite sure on requirement yet. Do we really want real
time for flow aggregated data or some fine-grained time interval (like 15 secs) should be
good enough - if we want to show some nice metrics chart for flow, this should be fine. Even
for real time, we don't have to aggregate everything from raw entity table, we don't have
to duplicated count metrics again for finished apps. Isn't it?

bq. (3) time-based flow aggregation: This is different than the online aggregation in the
sense that it is aggregated along the time boundary (e.g. “daily”, “weekly”, etc.).
This can be based on the Phoenix schema. This can be populated in an offline fashion (e.g.
running a mapreduce job).
Any special reason not to handle it in the same way above - as HBase coprocessor? It just
sound like gross-grained time interval. Isn't it?

bq. This is another “offline” aggregation type. Also, I believe we’re talking about
only time-based aggregation. In other words, we would aggregate values for users only with
a well-defined time window. There won’t be a “real-time” aggregation of values, similar
to the flow aggregation.
I would also call for a fine-grained time interval (closed to real-time) because the aggregated
resource metrics on user could be used in billing hadoop usage in a shared environment (no
matter private or public cloud), so user need to know more details on resource consumption
especially in some random peak time.

bq. Very much agree with separation into 2 categories "online" versus "periodic". I think
this will be natural split between the native HBase tables for the former and the Phoenix
approach for the latter to each emphasize their relative strengths.
I would question the necessary for "online" again if this mean "real time" instead of fine-grained
time interval. Actually, as a building block, every container metrics (cpu, memory, etc.)
are generated in a time interval instead of real time. As a result, we never know the exactly
snapshot of whole system in a precisely time but only can try to getting closer.

> [Aggregation] Application/Flow/User/Queue Level Aggregations
> ------------------------------------------------------------
>                 Key: YARN-3815
>                 URL: https://issues.apache.org/jira/browse/YARN-3815
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Junping Du
>            Assignee: Junping Du
>            Priority: Critical
>         Attachments: Timeline Service Nextgen Flow, User, Queue Level Aggregations (v1).pdf
> Per previous discussions in some design documents for YARN-2928, the basic scenario is
the query for stats can happen on:
> - Application level, expect return: an application with aggregated stats
> - Flow level, expect return: aggregated stats for a flow_run, flow_version and flow 
> - User level, expect return: aggregated stats for applications submitted by user
> - Queue level, expect return: aggregated stats for applications within the Queue
> Application states is the basic building block for all other level aggregations. We can
provide Flow/User/Queue level aggregated statistics info based on application states (a dedicated
table for application states is needed which is missing from previous design documents like
HBase/Phoenix schema design). 

This message was sent by Atlassian JIRA

View raw message