hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: Pluggable block index
Date Mon, 06 Jun 2011 06:33:42 GMT
Ok, the block index is only storing the first key of each block?
Hmm... I think we can store a pointer to an exact position in the
block, or at least allow that (for the FST implementation).

How efficient is the current seeking?

> I have previously thought about prefix compression, it seemed doable,

It does look like prefix compression should be doable.  Eg, we'd seek
to a position based on the block index (from which we'd have the
entire key).  From the seek'd to position, we could scan and load up
each subsequent prefix compressed key into a KeyValue, though right
the KV wouldn't be 'pointing' back to the internals of the block, it'd
be creating a whole new byte[] for each KV (which could have it's own
garbage related ramifications).

> you'd need a compressing algorithm

Lucene's terms dict is very simple.  The next key has the position at
which the previous key differs.

On Sat, Jun 4, 2011 at 3:35 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
> Also, dont break it :-)
> Part of the goal of HFile was to build something quick and reliable.
> It can be hard to know you have all the corner cases down and you
> won't find out in 6 months that every single piece of data you have
> put in HBase is corrupt.  Keeping it simple is one strategy.
> I have previously thought about prefix compression, it seemed doable,
> you'd need a compressing algorithm, then in the Scanner you would
> expand KeyValues and callers would end up with copies, not views on,
> the original data.  The JVM is fairly good about short lived objects
> (up to a certain allocation rate that is), and while the original goal
> was to reduce memory usage, it could make sense to take a higher short
> term allocation rate if the wins from prefix compression are there.
> Also note that in whole-system profiling, often repeated methods in
> KeyValue do pop up.  The goal of KeyValue was to have a format that
> didnt require deserialization into larger data structures (hence the
> lack of vint), and would be simple and fast.  Undoing that work should
> be accompanied with profiling evidence that new slowdowns were not
> introduced.
> -ryan
> On Sat, Jun 4, 2011 at 3:30 PM, Jason Rutherglen
> <jason.rutherglen@gmail.com> wrote:
>>> You'd have to change how the Scanner code works, etc.  You'll find out.
>> Nice!  Sounds fun.
>> On Sat, Jun 4, 2011 at 3:27 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>> What are the specs/goals of a pluggable block index?  Right now the
>>> block index is fairly tied deep in how HFile works. You'd have to
>>> change how the Scanner code works, etc.  You'll find out.
>>> On Sat, Jun 4, 2011 at 3:17 PM, Stack <saint.ack@gmail.com> wrote:
>>>> I do not know of one.  FYI hfile is pretty standalone regards tests etc.
 There is even a perf testing class for hfile
>>>> On Jun 4, 2011, at 14:44, Jason Rutherglen <jason.rutherglen@gmail.com>
>>>>> I want to take a wh/hack at creating a pluggable block index, is there
>>>>> an open issue for this?  I looked and couldn't find one.

View raw message