hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Will all HFiles managed by a regionserver kept open
Date Thu, 13 Jan 2011 06:57:23 GMT
retrieving data from disk is the most dominant element, until you are
fully cached in which case other factors inside the regionserver
become dominant. at this point copying memory, gc, algorithmic
complexity, etc become important.

On Wed, Jan 12, 2011 at 10:54 PM, Tao Xie <xietao.mailbox@gmail.com> wrote:
> Thanks for your response, Stack. I have a further question when
> understanding hbase.
> In my minds, I think a get is executed in the following process.
>
> hbase client <=> RS <=> DN
>
> 1) hbase client finds the RS managing the key; 2) RS knows the hfile and
> fetches data from DataNode, this may be a pread + scanning in the hbase data
> block; 3) record result is returned to client.
>
> Is this correct? So the most expensive operation is step 2?  Any other
> time-consuming places?
>
>
> 2011/1/13 Stack <stack@duboce.net>
>
>> Yes, all files are opened on startup and kept open.  Open of an hbase
>> storefile/hfile includes loading up of the file index and metadata.
>> In our experience, this overhead has been small.  Its currently not
>> accounted for in our general memory-counting.  We should for sure add
>> it.
>>
>> St.Ack
>>
>> On Wed, Jan 12, 2011 at 7:51 PM, Tao Xie <xietao.mailbox@gmail.com> wrote:
>> > hi, I know generally regionserver manages HRegions and in the HDFS layer
>> > data in HRegion are stored as HFile format. I want to know whether HFiles
>> > are all open and things lke block index are all loaded first to improve
>> > lookup performance? If so, what will happen if exceeding memory limit?
>> >
>> > Thanks.
>> >
>>
>

Mime
View raw message