hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vrushali C (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle
Date Fri, 31 Mar 2017 05:32:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950347#comment-15950347

Vrushali C commented on YARN-6375:

Hello Varun,

I am trying to understand the jira and I have some questions. I understand what you are saying
by aggregation could aggregate only those metrics that are reported by entities in the current
timeframe. I am trying to think of an example where we need aggregation for only currently
reported metrics vs aggregation for lifetime metrics.

CPU may be a good example of aggregating currently reported metrics? For instance, we might
want to see what is the CPU usage while these containers are running but once a container
is done, we need not consider it's last reported CPU for current CPU usage of this app. 

But, for other metrics, like say HDFS_BYTES_READ, perhaps we do want to think about how many
bytes have been read across tasks? I am thinking out loud. 

Also, for CPU, perhaps we might want to know across this application what was the cpu used
by all the containers in the lifetime of this app? What metrics are you thinking of.. 

> App level aggregation should not consider metric values reported in the previous aggregation
> --------------------------------------------------------------------------------------------------
>                 Key: YARN-6375
>                 URL: https://issues.apache.org/jira/browse/YARN-6375
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-6375-YARN-5355.01.patch
> Currently app level aggregation is done every 15 seconds.
> And we consider last reported metric value for each entity belonging to an app for aggregation.
> We however merely update the corresponding metric values for the entity on put. We never
remove the entries.
> But it is possible that multiple entities finish during lifetime of an application. We
however continue to consider them till the end.
> We should however not consider metric values of entities unless reported within the 15
second period.
> Consider containers. For a long running app, several containers would start and end at
various times during the lifetime of an app.
> To consider metrics for all the containers throughout the lifetime of app, hence wont
be correct.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message