hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Hash indexing of HFiles
Date Sat, 16 Jul 2011 20:08:49 GMT
On Fri, Jul 15, 2011 at 10:06 AM, Claudio Martella
<claudio.martella@tis.bz.it> wrote:
> On 7/15/11 6:24 PM, Stack wrote:
>> How do you figure the N in the below Claudio?
> N is the total amount of pairs in the sequence file. You know that when
> you finish flushing a memstore or compacting files.

So a perfect index?  If this is what you mean, won't the index often
be often be close in size to the data size? (And index lives in memory
at the moment; we have yet to make it so unused indices are let go if
unused.  And if this is what you mean, you can have a perfect index
now by setting hfile block size < your smallest cell size.


> That is interesting, the principle looks like the same as in HFile
> (blocks and Btree) but I have to understand the difference further.
>

In HFile, the index is for every ~64k blocks or so; one block could
have one entry only while the next block could have hundreds of
entries; either has one entry in index only.  So, its not the same as
mapfile.  It will write an index entry every 128 (IIRC) entries
whether the entries are 10M each or 10bytes each (So, the indices
could be wildly different in size based off cell size).

>> IIRC, we have
>> performance evaluation for various file types (You might be interested
>> in this recent posting by Mikhail Bautin of an hfile v2).
> I'd be interested in that, do you have a reference to it?
>

Here is Mikhail's post of an hfile v2
https://issues.apache.org/jira/browse/HBASE-3857

The HFile perf test tool is in code base at
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
 (It was used comparing hfile vs tfile vs mapfile and then hfile
tweaks; it may be a little stale).

Thanks for bringing up this topic Claudio and for taking the time to
do a bit of research.

St.Ack

Mime
View raw message