hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: a question storefileIndexSize
Date Wed, 25 May 2011 21:20:27 GMT
Good point Matt.  I forgot about compression.  Let me add not to the
above referenced section in the book....
St.Ack

On Wed, May 25, 2011 at 7:47 AM, Matt Corgan <mcorgan@hotpads.com> wrote:
> I have a table that compresses by 30x using gzip, so the default block size
> of 64 KB was writing 2 KB blocks to disk.  To reduce storefileIndexSize, I
> raised the block size to 256 KB, presumably writing ~8KB disk blocks which
> is still pretty small.  Maybe you could go even higher depending on your
> compression ratio.
>
> btw - why 10394 regions with only 3103 storefiles?
>
>
> 2011/5/25 Gaojinchao <gaojinchao@huawei.com>
>
>> Region size is 512M
>>
>> hbase.regionserver.handler.count 50
>> hbase.regionserver.global.memstore.upperLimit 0.4
>> hbase.regionserver.global.memstore.lowerLimit 0.35
>> hbase.hregion.memstore.flush.size 128M
>> hbase.hregion.max.filesize 512M
>> hbase.client.scanner.caching 1 hfile.block.cache.size 0.2
>>  hbase.hregion.memstore.block.multiplier 3
>> hbase.hstore.blockingStoreFiles 10
>> hbase.hstore.compaction.min.size 64M
>>
>> compress: gz
>>
>> dfs.block.size 256M
>>
>> -----邮件原件-----
>> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
>> 发送时间: 2011年5月25日 11:57
>> 收件人: user@hbase.apache.org
>> 主题: Re: a question storefileIndexSize
>>
>> 2011/5/24 Gaojinchao <gaojinchao@huawei.com>:
>> > Stack, Thanks for your reply.
>> > block size is default.
>> > My Key length is 26 bytes and value is 300~400 bytes.
>> > Is it big keys and small values ?
>> >
>>
>> Looks like you have 'small' keys.
>>
>> It looks like the index is about 1MB per storefile (storefiles=3103,
>> storefileIndexSize=3717).  Does this seem about right?  What size are
>> your regions?
>>
>> St.Ack
>>
>

Mime
View raw message