Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <1881008833.1242833685707.JavaMail.jira@brutus>
Date: Wed, 20 May 2009 08:34:45 -0700 (PDT)
From: "Evgeny Ryabitskiy (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Updated: (HBASE-1017) Region balancing does not bring newly
 added node within acceptable range
In-Reply-To: <1929845832.1227323564259.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HBASE-1017?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:all-tabpanel ]

Evgeny Ryabitskiy updated HBASE-1017:
-------------------------------------

    Attachment: HBASE-1017_v12_FINAL.patch

> Region balancing does not bring newly added node within acceptable range
> ------------------------------------------------------------------------
>
>                 Key: HBASE-1017
>                 URL: https://issues.apache.org/jira/browse/HBASE-1017
>             Project: Hadoop HBase
>          Issue Type: Improvement
>    Affects Versions: 0.19.0
>            Reporter: Jonathan Gray
>            Assignee: Evgeny Ryabitskiy
>            Priority: Minor
>             Fix For: 0.20.0
>
>         Attachments: HBASE-1017_v1.patch, HBASE-1017_v10.patch, HBASE-101=
7_v11_FINAL.patch, HBASE-1017_v12_FINAL.patch, HBASE-1017_v2.patch, HBASE-1=
017_v4.patch, HBASE-1017_v5.patch, HBASE-1017_v6.patch, HBASE-1017_v7.patch=
, HBASE-1017_v8.patch, HBASE-1017_v9.patch, loadbalance2.0.patch
>
>
> With a 10 node cluster, there were only 9 online nodes.  With about 215 t=
otal regions, each of the 9 had around 24 regions (average load is 24).  Sl=
op is 10% so 22 to 26 is the acceptable range.
> Starting up the 10th node, master log showed:
> {code}
> 2008-11-21 15:57:51,521 INFO org.apache.hadoop.hbase.master.ServerManager=
: Received start message from: 72.34.249.210:60020
> 2008-11-21 15:57:53,351 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Server 72.34.249.219:60020 is overloaded. Server load: 25 avg: 22.0, slo=
p: 0.1
> 2008-11-21 15:57:53,351 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Choosing to reassign 3 regions. mostLoadedRegions has 10 regions in it.
> 2008-11-21 15:57:53,351 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Going to close region streamitems,^@^@^@^@^AH=C3=AF=C2=BF=C2=BD;,1225411=
051632
> 2008-11-21 15:57:53,351 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Going to close region streamitems,^@^@^@^@^@=C3=AF=C2=BF=C2=BD=C3=9D,122=
5411056686
> 2008-11-21 15:57:53,351 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Going to close region groups,,1222913580957
> 2008-11-21 15:57:53,975 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Server 72.34.249.213:60020 is overloaded. Server load: 25 avg: 22.0, slo=
p: 0.1
> 2008-11-21 15:57:53,975 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Choosing to reassign 3 regions. mostLoadedRegions has 10 regions in it.
> 2008-11-21 15:57:53,976 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Going to close region upgrade,,1226892014784
> 2008-11-21 15:57:53,976 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Going to close region streamitems,^@^@^@^@^@3^Z=C3=AF=C2=BF=C2=BD,122541=
1056701
> 2008-11-21 15:57:53,976 DEBUG org.apache.hadoop.hbase.master.RegionManage=
r: Going to close region streamitems,^@^@^@^@^@         ^L,1225411049042
> {code}
> The new regionserver received only 6 regions.  This happened because when=
 the 10th came in, average load dropped to 22.  This caused two servers wit=
h 25 regions (acceptable when avg was 24 but not now) to reassign 3 of thei=
r regions each to bring them back down to the average.  Unfortunately all o=
ther regions remained within the 10% slop (20 to 24) so they were not overl=
oaded and thus did not reassign off any regions.  It was only chance that m=
ade even 6 of the regions get reassigned as there could have been exactly 2=
4 on each server, in which case none would have been assigned to the new no=
de.
> This will behave worse on larger clusters when adding a new node has litt=
le impact on the avg load/server.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.