Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Vladimir Rodionov <vrodionov@carrieriq.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Date: Wed, 26 Oct 2011 16:31:29 -0700
Subject: RE: Random I/O performance
Thread-Topic: Random I/O performance
Thread-Index: AcyUNPZZCXpAXt94Qt+4gt9sIQf4iQAAmy+b
Message-ID: 
 <DC5EBE7F3610EB4CA5C7E92D76873E15170BE3E361@exchange2007.carrieriq.com>
References: 
 <DC5EBE7F3610EB4CA5C7E92D76873E15170BE3E35B@exchange2007.carrieriq.com>
	<CADcMMgHCcNALQKfB85YnVBPaCTd1zNXq3rKQkbVL1=Qx5QOWHg@mail.gmail.com>
	<DC5EBE7F3610EB4CA5C7E92D76873E15170BE3E35C@exchange2007.carrieriq.com>
	<CADcMMgErsVnTgbarSa49rAAKM+2-3DuQDkM-PokkhaZvGzUh5A@mail.gmail.com>
	<DC5EBE7F3610EB4CA5C7E92D76873E15170BE3E35F@exchange2007.carrieriq.com>
	<CADcMMgHj9bk93Tbj3yOYJzJVgm7=tYN4upF9yoA5gooUAiOVUQ@mail.gmail.com>,<CALte62zaJ5Scj3ukF35iBE=2=F=Paen59n1sXF2NkTTWe9LzBg@mail.gmail.com>
In-Reply-To: 
 <CALte62zaJ5Scj3ukF35iBE=2=F=Paen59n1sXF2NkTTWe9LzBg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Great

Why is nobody using it?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: Wednesday, October 26, 2011 4:13 PM
To: dev@hbase.apache.org
Subject: Re: Random I/O performance

>> Off-heap cache is experimental in 0.92 and TRUNK.
As of now, TestSlabCache passes consistently in 0.92 and TRUNK.

Li Pi's slides from Aug can be found here:
https://docs.google.com/present/view?id=3Dd23xkzr_55hgnvngf6
Toward the end of it, you can find performance chart.

On Wed, Oct 26, 2011 at 3:49 PM, Stack <stack@duboce.net> wrote:

> On Wed, Oct 26, 2011 at 2:50 PM, Vladimir Rodionov
> <vrodionov@carrieriq.com> wrote:
> >>> Are you hitting cache at all?
> >>
> >> Its totally random, due to the proposed key design which favored fast
> inserts. Keys are randomized
> >> values, that is why there is no data locality in row look ups. Effect =
of
> the cache (LruBlockCache?) is negligible
> >> in this case.
> >>
> >
> >>>So a different schema would get cache into the mix?
> >
> > You can/t change schema while system is in production
> >
>
> True but caveat Ted's note and FB fellas apparently did it three times
> before they hit on the 'right' schema (Not sure whether they took the
> portion being modified offline when changing schema)
>
> >
> >>>Its going to keep growing without bound?
> >
> >
> > No, we keep data for XX days than purge stale data from the table.
> >
> >
> > My question was: what else besides obvious -run all in parallel - can
> help to improve random I/O?
> >
> > 1. Will BLOOM filter help to optimize HBase Read path?
>
> Yes.  0.92 blooms will be less expensive than those in 0.90 (because
> the blooms are tiered and live in the LRU in 0.92 so they are let go
> if unused).
>
>
> > 2. We use compression already.
> > 3. Block size - does it really matter much?
>
> Not much in my experience.  Smaller blocks can help a little at the
> cost of some bloat in index size (Again 0.92 is better here because
> indices are partitioned and now also are in the LRU rather than pegged
> in RAM as they are in 0.90).
>
> > 4. Off heap block cache? Its in 92 trunk? Have anybody performed real
> performance tests on Off heap cache?
> >
>
> Off-heap cache is experimental in 0.92 and TRUNK.
>
> St.Ack
>

Confidentiality Notice:  The information contained in this message, includi=
ng any attachments hereto, may be confidential and is intended to be read o=
nly by the individual or entity to whom this message is addressed. If the r=
eader of this message is not the intended recipient or an agent or designee=
 of the intended recipient, please note that any review, use, disclosure or=
 distribution of this message or its attachments, in any form, is strictly =
prohibited.  If you have received this message in error, please immediately=
 notify the sender and/or Notifications@carrieriq.com and delete or destroy=
 any copy of this message and its attachments.