hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12311) Version stats in HFiles?
Date Wed, 22 Oct 2014 20:24:35 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180475#comment-14180475

Lars Hofhansl commented on HBASE-12311:

In the end, I'm mostly looking for alternatives to speed up (re)seek. The core of the problem
is that we always seek even if the seek will land us on the same block and hence the seek
is actually unnecessary. There is a check for that (see HFileReaderV2.reseekTo), but that
is too far down and we still need to recompare at the Store/SQM level.

Having some simple stats about number of versions per column and/or columns per row would
be a reasonable proxy. Ideally we'd need a need good guess on whether the seek would seek
us out of the current bock with high likelihood. If so we seek if not we do next() a few time.

So maybe a better stat would the number of bytes per column and per row. I.e. we'd sum up
the sizes of the KVs of version for a column and all the KVs for a row and than keep max/avg
stats about that. Then with knowing the HFile block size we can guess whether a seek to a
column or row would propel us out of the block or not.

With that is mind the metrics to keep would be something like avg/max COL_SIZE and avg/max
ROW_SIZE. The former would inform whether we should SEEK_NEXT_COL the latter whether we should
SEEK_NEXT_ROW - both instead of performing a series of SKIP.

Presumably for most use cases the distribution of the col and row size would be pretty similar
across the entire table (I know we have a bunch of such use cases).

> Version stats in HFiles?
> ------------------------
>                 Key: HBASE-12311
>                 URL: https://issues.apache.org/jira/browse/HBASE-12311
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: Lars Hofhansl
> In HBASE-9778 I basically punted the decision on whether doing repeated scanner.next()
called instead of the issueing (re)seeks to the user.
> I think we can do better.
> One way do that is maintain simple stats of what the maximum number of versions we've
seen for any row/col combination and store these in the HFile's metadata (just like the timerange,
oldest Put, etc).
> Then we estimate fairly accurately whether we have to expect lots of versions (i.e. seek
between columns is better) or not (in which case we'd issue repeated next()'s).

This message was sent by Atlassian JIRA

View raw message