hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joep Rottinghuis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4053) Change the way metric values are stored in HBase Storage
Date Fri, 13 Nov 2015 22:45:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004842#comment-15004842
] 

Joep Rottinghuis commented on YARN-4053:
----------------------------------------

Now that I'm looking at it again in detail with [~vrushalic] the perf impact is probably not
quite as bad. I thought for a moment that the iteration had to happen for each cell, but it
is needed for each column only.

I agree with your point that we should keep it generic and be future proof. If we have the
converter, then we should just use it. However, in FlowScanner.compareCellValues (line 357)
you're assuming a long and just blindly cast to long. If this happens to be a GenericObjectMapper
then that will fail.

Perhaps we should do the following: make converter extend comparable and also add a method
to sum two (or more) byte values in a vararg style. For the long implementation that should
simply be a regular sum, for the generic object mapper it may be a string append, or whatever
makes sense. Then we're future proof when another style of conversion and summing and comparison
can happen safely right?

Would that make sense?

> Change the way metric values are stored in HBase Storage
> --------------------------------------------------------
>
>                 Key: YARN-4053
>                 URL: https://issues.apache.org/jira/browse/YARN-4053
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>              Labels: yarn-2928-1st-milestone
>         Attachments: YARN-4053-YARN-2928.01.patch, YARN-4053-YARN-2928.02.patch, YARN-4053-feature-YARN-2928.03.patch
>
>
> Currently HBase implementation uses GenericObjectMapper to convert and store values in
backend HBase storage. This converts everything into a string representation(ASCII/UTF-8 encoded
byte array).
> While this is fine in most cases, it does not quite serve our use case for metrics. 
> So we need to decide how are we going to encode and decode metric values and store them
in HBase.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message