hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Pluggable block index
Date Mon, 06 Jun 2011 06:37:25 GMT
On Sun, Jun 5, 2011 at 11:33 PM, Jason Rutherglen
<jason.rutherglen@gmail.com> wrote:
> Ok, the block index is only storing the first key of each block?
> Hmm... I think we can store a pointer to an exact position in the
> block, or at least allow that (for the FST implementation).

Are you sure that is a good idea?  Surely the disk seeks would destroy
you on index load?



>
> How efficient is the current seeking?
>
>> I have previously thought about prefix compression, it seemed doable,
>
> It does look like prefix compression should be doable.  Eg, we'd seek
> to a position based on the block index (from which we'd have the
> entire key).  From the seek'd to position, we could scan and load up
> each subsequent prefix compressed key into a KeyValue, though right
> the KV wouldn't be 'pointing' back to the internals of the block, it'd
> be creating a whole new byte[] for each KV (which could have it's own
> garbage related ramifications).
>
>> you'd need a compressing algorithm
>
> Lucene's terms dict is very simple.  The next key has the position at
> which the previous key differs.
>
> On Sat, Jun 4, 2011 at 3:35 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>> Also, dont break it :-)
>>
>> Part of the goal of HFile was to build something quick and reliable.
>> It can be hard to know you have all the corner cases down and you
>> won't find out in 6 months that every single piece of data you have
>> put in HBase is corrupt.  Keeping it simple is one strategy.
>>
>> I have previously thought about prefix compression, it seemed doable,
>> you'd need a compressing algorithm, then in the Scanner you would
>> expand KeyValues and callers would end up with copies, not views on,
>> the original data.  The JVM is fairly good about short lived objects
>> (up to a certain allocation rate that is), and while the original goal
>> was to reduce memory usage, it could make sense to take a higher short
>> term allocation rate if the wins from prefix compression are there.
>>
>> Also note that in whole-system profiling, often repeated methods in
>> KeyValue do pop up.  The goal of KeyValue was to have a format that
>> didnt require deserialization into larger data structures (hence the
>> lack of vint), and would be simple and fast.  Undoing that work should
>> be accompanied with profiling evidence that new slowdowns were not
>> introduced.
>>
>> -ryan
>>
>> On Sat, Jun 4, 2011 at 3:30 PM, Jason Rutherglen
>> <jason.rutherglen@gmail.com> wrote:
>>>> You'd have to change how the Scanner code works, etc.  You'll find out.
>>>
>>> Nice!  Sounds fun.
>>>
>>> On Sat, Jun 4, 2011 at 3:27 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>>> What are the specs/goals of a pluggable block index?  Right now the
>>>> block index is fairly tied deep in how HFile works. You'd have to
>>>> change how the Scanner code works, etc.  You'll find out.
>>>>
>>>>
>>>>
>>>> On Sat, Jun 4, 2011 at 3:17 PM, Stack <saint.ack@gmail.com> wrote:
>>>>> I do not know of one.  FYI hfile is pretty standalone regards tests
etc.  There is even a perf testing class for hfile
>>>>>
>>>>>
>>>>>
>>>>> On Jun 4, 2011, at 14:44, Jason Rutherglen <jason.rutherglen@gmail.com>
wrote:
>>>>>
>>>>>> I want to take a wh/hack at creating a pluggable block index, is
there
>>>>>> an open issue for this?  I looked and couldn't find one.
>>>>>
>>>>
>>>
>>
>

Mime
View raw message