hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Corgan <mcor...@hotpads.com>
Subject Re: a question storefileIndexSize
Date Wed, 25 May 2011 15:09:15 GMT
also - how long are your column family name and column qualifiers?  they are
added to each row key in the index, so you want to make them as short as
possible


On Wed, May 25, 2011 at 10:47 AM, Matt Corgan <mcorgan@hotpads.com> wrote:

> I have a table that compresses by 30x using gzip, so the default block size
> of 64 KB was writing 2 KB blocks to disk.  To reduce storefileIndexSize, I
> raised the block size to 256 KB, presumably writing ~8KB disk blocks which
> is still pretty small.  Maybe you could go even higher depending on your
> compression ratio.
>
> btw - why 10394 regions with only 3103 storefiles?
>
>
>
> 2011/5/25 Gaojinchao <gaojinchao@huawei.com>
>
>> Region size is 512M
>>
>> hbase.regionserver.handler.count 50
>> hbase.regionserver.global.memstore.upperLimit 0.4
>> hbase.regionserver.global.memstore.lowerLimit 0.35
>> hbase.hregion.memstore.flush.size 128M
>> hbase.hregion.max.filesize 512M
>> hbase.client.scanner.caching 1 hfile.block.cache.size 0.2
>>  hbase.hregion.memstore.block.multiplier 3
>> hbase.hstore.blockingStoreFiles 10
>> hbase.hstore.compaction.min.size 64M
>>
>> compress: gz
>>
>> dfs.block.size 256M
>>
>> -----邮件原件-----
>> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
>> 发送时间: 2011年5月25日 11:57
>> 收件人: user@hbase.apache.org
>> 主题: Re: a question storefileIndexSize
>>
>> 2011/5/24 Gaojinchao <gaojinchao@huawei.com>:
>> > Stack, Thanks for your reply.
>> > block size is default.
>> > My Key length is 26 bytes and value is 300~400 bytes.
>> > Is it big keys and small values ?
>> >
>>
>> Looks like you have 'small' keys.
>>
>> It looks like the index is about 1MB per storefile (storefiles=3103,
>> storefileIndexSize=3717).  Does this seem about right?  What size are
>> your regions?
>>
>> St.Ack
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message