hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Pluggable block index
Date Mon, 06 Jun 2011 07:11:34 GMT
When I thought about it, I didn't think cross-block compression would
be a good idea - this is because you want to be able to decompress
each block independently of each other.  Perhaps a master HFile
dictionary or something.

-ryan

On Mon, Jun 6, 2011 at 12:06 AM, M. C. Srivas <mcsrivas@gmail.com> wrote:
> On Sun, Jun 5, 2011 at 11:37 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> On Sun, Jun 5, 2011 at 11:33 PM, Jason Rutherglen
>> <jason.rutherglen@gmail.com> wrote:
>> > Ok, the block index is only storing the first key of each block?
>> > Hmm... I think we can store a pointer to an exact position in the
>> > block, or at least allow that (for the FST implementation).
>>
>> Are you sure that is a good idea?  Surely the disk seeks would destroy
>> you on index load?
>>
>
> I agree, it would be pretty bad.
>
> But, assuming that the block size is set appropriately, copying one key per
> 100 or so values into the block index does not really bloat the hfile and is
> good trade-off to avoid the seeking. Plus, it does not prevent
> prefix-compression inside the block itself. Are we considering
> prefix-compression of keys across blocks?
>
>
>
>>
>>
>> >
>> > How efficient is the current seeking?
>> >
>> >> I have previously thought about prefix compression, it seemed doable,
>> >
>> > It does look like prefix compression should be doable.  Eg, we'd seek
>> > to a position based on the block index (from which we'd have the
>> > entire key).  From the seek'd to position, we could scan and load up
>> > each subsequent prefix compressed key into a KeyValue, though right
>> > the KV wouldn't be 'pointing' back to the internals of the block, it'd
>> > be creating a whole new byte[] for each KV (which could have it's own
>> > garbage related ramifications).
>> >
>> >> you'd need a compressing algorithm
>> >
>> > Lucene's terms dict is very simple.  The next key has the position at
>> > which the previous key differs.
>> >
>> > On Sat, Jun 4, 2011 at 3:35 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>> >> Also, dont break it :-)
>> >>
>> >> Part of the goal of HFile was to build something quick and reliable.
>> >> It can be hard to know you have all the corner cases down and you
>> >> won't find out in 6 months that every single piece of data you have
>> >> put in HBase is corrupt.  Keeping it simple is one strategy.
>> >>
>> >> I have previously thought about prefix compression, it seemed doable,
>> >> you'd need a compressing algorithm, then in the Scanner you would
>> >> expand KeyValues and callers would end up with copies, not views on,
>> >> the original data.  The JVM is fairly good about short lived objects
>> >> (up to a certain allocation rate that is), and while the original goal
>> >> was to reduce memory usage, it could make sense to take a higher short
>> >> term allocation rate if the wins from prefix compression are there.
>> >>
>> >> Also note that in whole-system profiling, often repeated methods in
>> >> KeyValue do pop up.  The goal of KeyValue was to have a format that
>> >> didnt require deserialization into larger data structures (hence the
>> >> lack of vint), and would be simple and fast.  Undoing that work should
>> >> be accompanied with profiling evidence that new slowdowns were not
>> >> introduced.
>> >>
>> >> -ryan
>> >>
>> >> On Sat, Jun 4, 2011 at 3:30 PM, Jason Rutherglen
>> >> <jason.rutherglen@gmail.com> wrote:
>> >>>> You'd have to change how the Scanner code works, etc.  You'll find
>> out.
>> >>>
>> >>> Nice!  Sounds fun.
>> >>>
>> >>> On Sat, Jun 4, 2011 at 3:27 PM, Ryan Rawson <ryanobjc@gmail.com>
>> wrote:
>> >>>> What are the specs/goals of a pluggable block index?  Right now
the
>> >>>> block index is fairly tied deep in how HFile works. You'd have to
>> >>>> change how the Scanner code works, etc.  You'll find out.
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Sat, Jun 4, 2011 at 3:17 PM, Stack <saint.ack@gmail.com>
wrote:
>> >>>>> I do not know of one.  FYI hfile is pretty standalone regards
tests
>> etc.  There is even a perf testing class for hfile
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Jun 4, 2011, at 14:44, Jason Rutherglen <
>> jason.rutherglen@gmail.com> wrote:
>> >>>>>
>> >>>>>> I want to take a wh/hack at creating a pluggable block index,
is
>> there
>> >>>>>> an open issue for this?  I looked and couldn't find one.
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>

Mime
View raw message