hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vrodio...@carrieriq.com>
Subject RE: Random I/O performance
Date Wed, 26 Oct 2011 23:31:29 GMT

Why is nobody using it?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

From: Ted Yu [yuzhihong@gmail.com]
Sent: Wednesday, October 26, 2011 4:13 PM
To: dev@hbase.apache.org
Subject: Re: Random I/O performance

>> Off-heap cache is experimental in 0.92 and TRUNK.
As of now, TestSlabCache passes consistently in 0.92 and TRUNK.

Li Pi's slides from Aug can be found here:
Toward the end of it, you can find performance chart.

On Wed, Oct 26, 2011 at 3:49 PM, Stack <stack@duboce.net> wrote:

> On Wed, Oct 26, 2011 at 2:50 PM, Vladimir Rodionov
> <vrodionov@carrieriq.com> wrote:
> >>> Are you hitting cache at all?
> >>
> >> Its totally random, due to the proposed key design which favored fast
> inserts. Keys are randomized
> >> values, that is why there is no data locality in row look ups. Effect of
> the cache (LruBlockCache?) is negligible
> >> in this case.
> >>
> >
> >>>So a different schema would get cache into the mix?
> >
> > You can/t change schema while system is in production
> >
> True but caveat Ted's note and FB fellas apparently did it three times
> before they hit on the 'right' schema (Not sure whether they took the
> portion being modified offline when changing schema)
> >
> >>>Its going to keep growing without bound?
> >
> >
> > No, we keep data for XX days than purge stale data from the table.
> >
> >
> > My question was: what else besides obvious -run all in parallel - can
> help to improve random I/O?
> >
> > 1. Will BLOOM filter help to optimize HBase Read path?
> Yes.  0.92 blooms will be less expensive than those in 0.90 (because
> the blooms are tiered and live in the LRU in 0.92 so they are let go
> if unused).
> > 2. We use compression already.
> > 3. Block size - does it really matter much?
> Not much in my experience.  Smaller blocks can help a little at the
> cost of some bloat in index size (Again 0.92 is better here because
> indices are partitioned and now also are in the LRU rather than pegged
> in RAM as they are in 0.90).
> > 4. Off heap block cache? Its in 92 trunk? Have anybody performed real
> performance tests on Off heap cache?
> >
> Off-heap cache is experimental in 0.92 and TRUNK.
> St.Ack

Confidentiality Notice:  The information contained in this message, including any attachments
hereto, may be confidential and is intended to be read only by the individual or entity to
whom this message is addressed. If the reader of this message is not the intended recipient
or an agent or designee of the intended recipient, please note that any review, use, disclosure
or distribution of this message or its attachments, in any form, is strictly prohibited. 
If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com
and delete or destroy any copy of this message and its attachments.

View raw message