hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Hbase reads an entire 64MB HDFS block over network when reading a single value?
Date Wed, 17 Sep 2008 21:08:35 GMT
No.  To get some random entry from a store file, the Mapfile index is
employed seeking the location of the asked-for key over in the remote file
in the fileystem.  See the MapFile#get function up in hadoop.  DFSClient
manages the work.  There are for sure inefficencies involved when the
fetched value is bytes but the block-size in HDFS is 64MB but its not the
case that the full block is pulled client-side to extract the wanted values
(nor the pulling of the complete file).

Its a bit of a hairy ride trying to hold on once you break below the surface
of (H)DFS Client. If you are about to deep-dive and want some company, feel
free keeping the discussion going up here on this list.


On Wed, Sep 17, 2008 at 5:08 AM, Cosmin Lehene <clehene@adobe.com> wrote:

> Hi,
> Does Hbase read the entire file from network (HDFS) when doing a get
> operation, or it's able to read just a smaller data segment?
> I got down to HStoreFile HBaseReader that does a MapFile.open, but I can't
> really figure what happens next...
> Thanks,
> Cosmin

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message