hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6435) [ATSv2] Can't retrieve more than 1000 versions of metrics in time series
Date Tue, 04 Apr 2017 08:10:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954779#comment-15954779

Varun Saxena commented on YARN-6435:

Thanks [~rohithsharma] for raising the issue.
A similar limit exists for Application Table too. So probably change should be made there

However, having said that if we increase max versions to max value of integer, it will directly
impact the storage footprint for the table and space required to store corresponding data.
Especially when it comes to entity table, the table can be quite large.
If a metric is printed every 10 second, this means we can at max retrieve a time series for
3 hours approximately which sounds quite less.

Maybe we can provide a configuration to control how many versions should we store for metric
column family with a higher suitable default?

[~vrushalic], your thoughts on the same?

> [ATSv2] Can't retrieve more than 1000 versions of metrics in time series
> ------------------------------------------------------------------------
>                 Key: YARN-6435
>                 URL: https://issues.apache.org/jira/browse/YARN-6435
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: timelineserver
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>            Priority: Critical
>         Attachments: YARN-6435.0001.patch
> It is observed that, even though *metricslimit* is set to 1500, maximum number of metrics
values retrieved is 1000. 
> This is due to, while creating EntityTable, metrics column family max version is specified
as 1000 which is hardcoded in {{EntityTable#DEFAULT_METRICS_MAX_VERSIONS}}. So, HBase will
return max version with following {{MIN(cf max version , user provided max version)}}. 
> This behavior is contradicting the documentation which claims that 
> {code}
> metricslimit - If specified, defines the number of metrics to return. Considered only
if fields contains METRICS/ALL or metricstoretrieve is specified. Ignored otherwise. The maximum
possible value for metricslimit can be maximum value of Integer. If it is not specified or
has a value less than 1, and metrics have to be retrieved, then metricslimit will be considered
as 1 i.e. latest single value of metric(s) will be returned.
> {code} 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message