hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
Date Mon, 26 Oct 2015 17:42:27 GMT

    [ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974623#comment-14974623

Varun Saxena commented on YARN-4053:

[~sjlee0], [~djp], [~vrushalic] and others, kindly review.
In this patch I simply assume that we will handle all metrics as longs. 
When AggregationOperation is SUM(which is added as a cell tag/attribute), on the coprocessor
side I assume this is for metrics because this is what it is meant for as of now. 
If we change this for something else, we can change the tag to indicate if this cell would
contain a long too.

Moreover, I have added a simple flag in ColumnHelper to indicate value has to be encoded and
decoded as a long. Once we add support for other metrics, maybe TimelineMetric should indicate
data type of metric but that is not being done for now so a flag(the approach adopted in patch)
should do, as of now.

Now there were a few other things which have to be handled as part of this patch but we have
not yet reached consensus on or discussed them.
# We need to decide how to indicate if a metric is to be aggregated or not. Currently this
kept part of column qualifier in YARN-3816. We can continue with that I guess. As MR job run
for offline aggregation would need this info as well.
# Decide which metric is a TIME_SERIES and which one is SINGLE_VALUE(get only the latest value).
Should we use tags for it and attach coprocessor with every table storing metrics ? Performance
implications if that's done ?

> Change the way metric values are stored in HBase Storage
> --------------------------------------------------------
>                 Key: YARN-4053
>                 URL: https://issues.apache.org/jira/browse/YARN-4053
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-4053-YARN-2928.01.patch, YARN-4053-YARN-2928.02.patch
> Currently HBase implementation uses GenericObjectMapper to convert and store values in
backend HBase storage. This converts everything into a string representation(ASCII/UTF-8 encoded
byte array).
> While this is fine in most cases, it does not quite serve our use case for metrics. 
> So we need to decide how are we going to encode and decode metric values and store them
in HBase.

This message was sent by Atlassian JIRA

View raw message