Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of magnito@gmail.com designates
 209.85.214.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=whw5pFovaRhsOxSn0GVVC6Hd/9EOOMQ0ESLL++W8AUqRJsZMRnZ+dF74doHmCQNKha
         JKJfHXMzgMq1tHeCPjOwuKzQkVBe44qD9g7II7meKjsfZRD4pUm+p+PW9LTvddpaIav6
         MQ1DP6lTTmcL7sGmMGHd0Rd1EaQkDX6njq0s0=
MIME-Version: 1.0
In-Reply-To: <AANLkTim+uPrcJirE_8ojPz06ZyO5zY_SKN5uX7LTXnLo@mail.gmail.com>
References: <AANLkTincdTWT2QTqBdY_JWVTmMAZ2CDcxfN4i1uq8OyB@mail.gmail.com>
	<965525.19357.qm@web65510.mail.ac4.yahoo.com>
	<AANLkTikt_ipjO1PWPQFEO3UiRhJ+qjqobTFn7KyjAFR5@mail.gmail.com>
	<AANLkTim+uPrcJirE_8ojPz06ZyO5zY_SKN5uX7LTXnLo@mail.gmail.com>
Date: Wed, 24 Nov 2010 11:53:12 -0800
Message-ID: <AANLkTikKLG-K3H111F1hDfm10M+1P6FS7gchVTqouOe2@mail.gmail.com>
Subject: Re: question about meta data query intensity
From: Jack Levin <magnito@gmail.com>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Yes, I am game.

-Jack

On Wed, Nov 24, 2010 at 11:02 AM, Stack <stack@duboce.net> wrote:
> If you are game for deploying an instrumented jar, we could log client
> lookups in .META. and try and figure if it profligate.
> St.Ack
>
> On Wed, Nov 24, 2010 at 9:25 AM, Jack Levin <magnito@gmail.com> wrote:
>> Yes, but that does not alleviate CPU contention should there be too
>> many queries to a single region server. =A0 On a separate topic, is
>> 'compression' in the works for REST gateway? =A0 Similar to
>> mysql_client_compression? =A0We plan to drop in 500K or more queries at
>> a time into the REST, and it would be interesting to see the
>> performance gain against uncompressed data.
>>
>> Thanks.
>>
>> -Jack
>>
>> On Wed, Nov 24, 2010 at 9:04 AM, Andrew Purtell <apurtell@apache.org> wr=
ote:
>>> The REST gateway (Stargate) is a long lived client. :-)
>>>
>>> It uses HTablePool internally so this will keep some warm table referen=
ces around in addition to the region location caching that HConnectionManag=
er does behind the scenes. (10 references, but this could be made configura=
ble.)
>>>
>>> Best regards,
>>>
>>> =A0 =A0- Andy
>>>
>>> --- On Tue, 11/23/10, Jack Levin <magnito@gmail.com> wrote:
>>>
>>>> From: Jack Levin <magnito@gmail.com>
>>>> Subject: Re: question about meta data query intensity
>>>> To: user@hbase.apache.org
>>>> Date: Tuesday, November 23, 2010, 11:06 AM
>>>> its REST, and generally no long lived
>>>> clients, yes, caching of regions
>>>> helps however, we expect long tail hits that will be
>>>> uncached, which
>>>> may stress out meta region, that being said, is it possible
>>>> create
>>>> affinity and nail meta region into a beefy server or set of
>>>> beefy
>>>> servers?
>>>>
>>>> -Jack
>>>>
>>>> On Tue, Nov 23, 2010 at 10:58 AM, Jonathan Gray <jgray@fb.com>
>>>> wrote:
>>>> > Are you going to have long-lived clients? =A0How are
>>>> you accessing HBase? =A0REST or Thrift gateways? =A0Caching of
>>>> region locations should help significantly so that it's only
>>>> a bottleneck right at the startup of the
>>>> cluster/gateways/clients.
>>>> >
>>>> >> -----Original Message-----
>>>> >> From: Jack Levin [mailto:magnito@gmail.com]
>>>> >> Sent: Tuesday, November 23, 2010 10:53 AM
>>>> >> To: user@hbase.apache.org
>>>> >> Subject: Re: question about meta data query
>>>> intensity
>>>> >>
>>>> >> my concern is that we plane to have 120
>>>> regionservers with 1000
>>>> >> Regions each, so the hits to meta could be quite
>>>> intense. =A0(why so
>>>> >> many regions? we are storing 1 Petabyte of data of
>>>> images into hbase).
>>>> >>
>>>> >> -Jack
>>>> >>
>>>> >> On Tue, Nov 23, 2010 at 9:50 AM, Jonathan Gray
>>>> <jgray@fb.com>
>>>> wrote:
>>>> >> > It is possible that it could be a bottleneck
>>>> but usually is not.
>>>> >> =A0Generally production HBase installations have
>>>> long-lived clients, so
>>>> >> the client-side caching is sufficient to reduce
>>>> the amount of load to
>>>> >> META (virtually 0 clean cluster is at steady-state
>>>> / no region
>>>> >> movement).
>>>> >> >
>>>> >> > For MapReduce, you do make new clients but
>>>> generally only need to
>>>> >> query for one region per task.
>>>> >> >
>>>> >> > It is not currently possible to split META.
>>>> =A0We hard-coded some stuff
>>>> >> a while back to make things easier and in the name
>>>> of correctness.
>>>> >> >
>>>> >> > HBASE-3171 is about removing the ROOT region
>>>> and putting the META
>>>> >> region(s) locations into ZK directly. =A0When we
>>>> make that change, we
>>>> >> could probably also re-enable the splitting of
>>>> META.
>>>> >> >
>>>> >> > JG
>>>> >> >
>>>> >> >> -----Original Message-----
>>>> >> >> From: Jack Levin [mailto:magnito@gmail.com]
>>>> >> >> Sent: Tuesday, November 23, 2010 9:31 AM
>>>> >> >> To: user@hbase.apache.org
>>>> >> >> Subject: question about meta data query
>>>> intensity
>>>> >> >>
>>>> >> >> Hello, I am curious if there is a
>>>> potential bottleneck in .META.
>>>> >> >> ownership by a single region server. =A0Is
>>>> it possible (safe) to split
>>>> >> >> meta region into several?
>>>> >> >>
>>>> >> >> -Jack
>>>> >> >
>>>> >
>>>>
>>>
>>>
>>>
>>>
>>
>