hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Corgan <mcor...@hotpads.com>
Subject Re: a question storefileIndexSize
Date Wed, 25 May 2011 23:49:42 GMT
I was thinking it would be a nice feature if each time an hfile was written
it kept a count of the raw bytes (before compression) to make it easy to
compare to the file size on disk.  It could report it in the web interface
next to the disk size.


2011/5/25 Stack <stack@duboce.net>

> Good point Matt.  I forgot about compression.  Let me add not to the
> above referenced section in the book....
> St.Ack
>
> On Wed, May 25, 2011 at 7:47 AM, Matt Corgan <mcorgan@hotpads.com> wrote:
> > I have a table that compresses by 30x using gzip, so the default block
> size
> > of 64 KB was writing 2 KB blocks to disk.  To reduce storefileIndexSize,
> I
> > raised the block size to 256 KB, presumably writing ~8KB disk blocks
> which
> > is still pretty small.  Maybe you could go even higher depending on your
> > compression ratio.
> >
> > btw - why 10394 regions with only 3103 storefiles?
> >
> >
> > 2011/5/25 Gaojinchao <gaojinchao@huawei.com>
> >
> >> Region size is 512M
> >>
> >> hbase.regionserver.handler.count 50
> >> hbase.regionserver.global.memstore.upperLimit 0.4
> >> hbase.regionserver.global.memstore.lowerLimit 0.35
> >> hbase.hregion.memstore.flush.size 128M
> >> hbase.hregion.max.filesize 512M
> >> hbase.client.scanner.caching 1 hfile.block.cache.size 0.2
> >>  hbase.hregion.memstore.block.multiplier 3
> >> hbase.hstore.blockingStoreFiles 10
> >> hbase.hstore.compaction.min.size 64M
> >>
> >> compress: gz
> >>
> >> dfs.block.size 256M
> >>
> >> -----邮件原件-----
> >> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> >> 发送时间: 2011年5月25日 11:57
> >> 收件人: user@hbase.apache.org
> >> 主题: Re: a question storefileIndexSize
> >>
> >> 2011/5/24 Gaojinchao <gaojinchao@huawei.com>:
> >> > Stack, Thanks for your reply.
> >> > block size is default.
> >> > My Key length is 26 bytes and value is 300~400 bytes.
> >> > Is it big keys and small values ?
> >> >
> >>
> >> Looks like you have 'small' keys.
> >>
> >> It looks like the index is about 1MB per storefile (storefiles=3103,
> >> storefileIndexSize=3717).  Does this seem about right?  What size are
> >> your regions?
> >>
> >> St.Ack
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message