hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3816) [Aggregation] App-level aggregation and accumulation for YARN system metrics
Date Thu, 17 Sep 2015 08:20:47 GMT

    [ https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791775#comment-14791775
] 

Varun Saxena commented on YARN-3816:
------------------------------------

[~djp],
{quote}
In TimelineMetric#accumulateTo, can latestMetric be TIME_SERIES ? If not(seems to be the case
as per current code), is the else part of the condition if (latestMetric.getType().equals(Type.SINGLE_VALUE))
{ required ? Wont be handling TIME_SERIES then ?
I am not sure if I understand your comments correctly. But it definitely support TIME_SERIES
type for latestMetrics and handle two types separately.
{quote}
Actually I should have worded my query differently. accumulateTo by itself can handle TIME_SERIES.
This is more from the context of caller. I am not sure if Li's patch is calling it but in
TimelineCollector#aggregateMetrics, we have code like below. Here, I see latestTimelineMetrics.retrieveSingleDataValue()
being called, which will throw an exception if metric type is not SINGLE_VALUE. Objective
of throwing exception here ? As we have to get a single value for delta calculations, for
TIME_SERIES maybe we can take value the latest timestamp value. 
I was getting confused by this code(calling method which throws exception for time series).
So was wondering if we wont be handling time series.
{code}
213	            TimelineMetric latestTimelineMetrics = entityIdMap.get(entityId);
214	
215	            Number delta = null;
216	            // new added metric for specific entityId
217	            if (latestTimelineMetrics == null) {
218	              delta = metric.retrieveSingleDataValue();
219	            } else {
220	              delta = TimelineMetricCalculator.sub(
221	                  metric.retrieveSingleDataValue(),
222	                  latestTimelineMetrics.retrieveSingleDataValue());
223	            }
...........................
250	              TimelineMetric newAggregatedArea = metric.accumulateTo(
251	                  oldAggregatedArea, latestTimelineMetrics, aggregatedTime,
252	                  TimelineMetric.Operation.SUM);
{code}

> [Aggregation] App-level aggregation and accumulation for YARN system metrics
> ----------------------------------------------------------------------------
>
>                 Key: YARN-3816
>                 URL: https://issues.apache.org/jira/browse/YARN-3816
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Junping Du
>            Assignee: Junping Du
>         Attachments: Application Level Aggregation of Timeline Data.pdf, YARN-3816-YARN-2928-v1.patch,
YARN-3816-YARN-2928-v2.1.patch, YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch,
YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.patch, YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch
>
>
> We need application level aggregation of Timeline data:
> - To present end user aggregated states for each application, include: resource (CPU,
Memory) consumption across all containers, number of containers launched/completed/failed,
etc. We need this for apps while they are running as well as when they are done.
> - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be aggregated to show
details of states in framework level.
> - Other level (Flow/User/Queue) aggregation can be more efficient to be based on Application-level
aggregations rather than raw entity-level data as much less raws need to scan (with filter
out non-aggregated entities, like: events, configurations, etc.).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message