Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of nspiegelberg@fb.com
 designates 67.231.153.30 as permitted sender)
From: Nicolas Spiegelberg <nspiegelberg@fb.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>,
        lars hofhansl
	<lhofhansl@yahoo.com>
Subject: Re: region size/count per regionserver
Thread-Topic: region size/count per regionserver
Thread-Index: AQHMmL9gG6UAJEr0FkKZD9aM0dQthJWYoaMAgAI9xQD//5c/AA==
Date: Thu, 3 Nov 2011 02:55:45 +0000
Message-ID: <CAD74DAE.2010A%nspiegelberg@fb.com>
In-Reply-To: <1320286238.35445.YahooMailNeo@web121702.mail.ne1.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.13.0.110805
Content-Type: text/plain; charset="us-ascii"
Content-ID: <697F268D76C8D441AFD7AA52D0DAD2D9@fb.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Region Scalability is definitely an investigation item that has not been
covered yet.  We solved the problem with horizontal sharding into multiple
clusters instead of tackling that subject with the timeframe we had.  I'm
guessing the 2-level ROOT/META was a response to that problem.  On the
actual region count / data size, that all depends on how high you want to
scale your StoreFile size.  10GB StoreFiles are currently normal /
reasonable.

On 11/2/11 7:10 PM, "lars hofhansl" <lhofhansl@yahoo.com> wrote:

>Do we know what would need to change in HBase in order to be able to
>manage more regions per regionserver?
>With 20 regions per server, one would need 300G regions to just utilize
>6T of drive space.
>
>
>To utilize a regionserver/datanode with 24T drive space the region size
>would be an insane 1T.
>
>-- Lars
>
>________________________________
>From: Nicolas Spiegelberg <nspiegelberg@fb.com>
>To: "user@hbase.apache.org" <user@hbase.apache.org>
>Cc: Karthik Ranganathan <kranganathan@fb.com>; Kannan Muthukkaruppan
><Kannan@fb.com>
>Sent: Tuesday, November 1, 2011 3:57 PM
>Subject: Re: region size/count per regionserver
>
>Simple answer
>-------------
>20 regions/server & <2000 regions/cluster is a good rule of thumb if you
>can't profile your workload yet.  You really want to ensure that
>
>1) You need to limits the regions/cluster so the master can have a
>reasonable startup time & can handle all the region state transitions via
>ZK.  Most bigger companies are running 2,000 in production and achieve
>reasonable startup times (< 2 minutes for region assignment on cold
>start).  If you want to test the scalability of that algorithm beyond what
>other companies need, admin beware.
>2) The more regions/server you have, the faster that recovery can happen
>after RS death because you can currently parallelize recovery on a
>region-granularity.  Too many regions/server and #1 starts to be a
>problem.
>
>
>
>Complicated answer
>------------------
>More information is optimize this formula.  Additional considerations:
>
>1) Are you IO-bound or CPU-bound
>2) What is your grid topology like
>3) What is your network hardware like
>4) How many disks (not just size)
>5) What is the data locality between RegionServer & DataNode
>
>In the Facebook case, we have 5 racks with 20 nodes each.  Servers in the
>rack are connected by 1G Eth to a switch with a 10G uplink.  We are
>network bound.  Our saturation point is mostly commonly on the top-of-rack
>switch.  With 20 regions/server, we can roughly parallelize our
>distributed log splitting within a single rack on RS death (although 2
>regions do split off-rack).  This minimizes top-of-rack traffic and
>optimized our recovery time.  Even if you are CPU-bound, log splitting
>(hence recovery time) is an IO-bound operation.  A lot of our work on
>region assignment is about maximizing data locality, even on RS death, so
>we avoid top-of-rack saturation.
>
>
>On 11/1/11 10:54 AM, "Sujee Maniyam" <sujee@sujee.net> wrote:
>
>>HI all,
>>My HBase cluster is 10 nodes, each node has 12core ,   48G RAM, 24TB
>>disk,
>>10GEthernet.
>>My region size is 1GB.
>>
>>Any guidelines on how many regions can a RS  handle comfortably?
>>I vaguely remember reading some where to have no more than 1000 regions /
>>server; that comes to 1TB / server.  Seems pretty low for the current
>>hardware config.
>>
>>Any rules of thumb?  experiences?
>>
>>thanks
>>Sujee
>>
>>http://sujee.net