hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Converting byte[] to ByteBuffer
Date Sat, 09 Jul 2011 02:05:39 GMT

When running on top of Mapr, hbase has fast cached access to locally stored
files, the Mapr client ensures that. Likewise, hdfs should also ensure that
local reads are fast and come out of cache as necessary. Eg: the kernel
block cache.

I wouldn't support mmap, it would require 2 different read path
implementations. You will never know when a read is not local.

Hdfs needs to provide faster local reads imo. Managing the block cache in
not heap might work but you also might get there and find the dbb accounting
overhead kills.
On Jul 8, 2011 6:47 PM, "Jason Rutherglen" <jason.rutherglen@gmail.com>
> There are couple of things here, one is direct byte buffers to put the
> blocks outside of heap, the other is MMap'ing the blocks directly from
> the underlying HDFS file.
> I think they both make sense. And I'm not sure MapR's solution will
> be that much better if the latter is implemented in HBase.
> On Fri, Jul 8, 2011 at 6:26 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>> The overhead in a byte buffer is the extra integers to keep track of the
>> mark, position, limit.
>> I am not sure that putting the block cache in to heap is the way to go.
>> Getting faster local dfs reads is important, and if you run hbase on top
>> Mapr, these things are taken care of for you.
>> On Jul 8, 2011 6:20 PM, "Jason Rutherglen" <jason.rutherglen@gmail.com>
>> wrote:
>>> Also, it's for a good cause, moving the blocks out of main heap using
>>> direct byte buffers or some other more native-like facility (if DBB's
>>> don't work).
>>> On Fri, Jul 8, 2011 at 5:34 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>>> Where? Everywhere? An array is 24 bytes, bb is 56 bytes. Also the API
>>>> is...annoying.
>>>> On Jul 8, 2011 4:51 PM, "Jason Rutherglen" <jason.rutherglen@gmail.com>
>>>> wrote:
>>>>> Is there an open issue for this? How hard will this be? :)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message