hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: question about RegionManager
Date Tue, 07 Sep 2010 19:57:11 GMT
On Mon, Sep 6, 2010 at 11:34 PM, Tao Xie <xietao.mailbox@gmail.com> wrote:

> But when I directly load data into HDFS using HDFS API, the disks are
> balanced.
> I use hadoop-0.20.2.
>
>
Yes, the bugs occur when processing a large volume of block deletions. See
HADOOP-5124 and HDFS-611. HBase's compactions cause a larger deletion rate
than typical HDFS usage.

-Todd


> 2010/9/7 Todd Lipcon <todd@cloudera.com>
>
> > On Mon, Sep 6, 2010 at 9:08 PM, Jonathan Gray <jgray@facebook.com>
> wrote:
> >
> > > You're looking at sizes on disk?  Then this has nothing to do with
> HBase
> > > load balancing.
> > >
> > > HBase does not move blocks around on the HDFS layer or deal with which
> > > physical disks are used, that is completely the responsibility of HDFS.
> > >
> > > Periodically HBase will perform major compactions on regions which
> causes
> > > data to be rewritten.  This creates new files so could change what is
> in
> > > HDFS.
> > >
> >
> > There are some bugs in HDFS in 0.20 which can create this out-of-balance
> > scenario.
> >
> > If you use CDH3b2 you should have a few patches which help to rectify the
> > situation, in particular HDFS-611.
> >
> > Thanks
> > -Todd
> >
> >
> > >
> > > JG
> > >
> > > > -----Original Message-----
> > > > From: Tao Xie [mailto:xietao.mailbox@gmail.com]
> > > > Sent: Monday, September 06, 2010 8:38 PM
> > > > To: user@hbase.apache.org
> > > > Subject: Re: question about RegionManager
> > > >
> > > > Actually, I'm a newbie of HBase. I went to read the code of assigning
> > > > region
> > > > because I met a load imbalance problem in my hbase cluster. I run 1+6
> > > > nodes
> > > > hbase cluster, 1 node as master & client, the other nodes as region
> > > > server
> > > > and data nodes. I run YCSB to insert records. In the inserting time,
> I
> > > > find
> > > > the data written to data nodes have different data size on disks.  I
> > > > think
> > > > HDFS is doing well in balancing write. So is this problem due to
> HBase?
> > > >
> > > > Btw, after finished writing for minutes, the disks get balanced
> > > > finally. I
> > > > think maybe there is a LoadBalance like deamon thread working on
> this.
> > > > Can
> > > > anyone explain this? Many thanks.
> > > >
> > > > After inserting 160M 1k records, my six datanodes are greatly
> > > > imbalanced.
> > > >
> > > > 10.1.0.125: /dev/sdb1             280G   89G  178G  34%
> /mnt/DP_disk1
> > > >
> > > > 10.1.0.125: /dev/sdc1             280G   91G  176G  35%
> /mnt/DP_disk2
> > > >
> > > > 10.1.0.125: /dev/sdd1             280G   91G  176G  34%
> /mnt/DP_disk3
> > > >
> > > > 10.1.0.121: /dev/sdb1             280G   15G  251G   6%
> /mnt/DP_disk1
> > > >
> > > > 10.1.0.121: /dev/sdc1             280G   16G  250G   6%
> /mnt/DP_disk2
> > > >
> > > > 10.1.0.121: /dev/sdd1             280G   15G  251G   6%
> /mnt/DP_disk3
> > > >
> > > > 10.1.0.122: /dev/sdb1             280G   15G  251G   6%
> /mnt/DP_disk1
> > > >
> > > > 10.1.0.122: /dev/sdc1             280G   15G  252G   6%
> /mnt/DP_disk2
> > > >
> > > > 10.1.0.122: /dev/sdd1             280G   13G  253G   5%
> /mnt/DP_disk3
> > > >
> > > > 10.1.0.124: /dev/sdb1             280G   14G  253G   5%
> /mnt/DP_disk1
> > > >
> > > > 10.1.0.124: /dev/sdc1             280G   15G  252G   6%
> /mnt/DP_disk2
> > > >
> > > > 10.1.0.124: /dev/sdd1             280G   14G  253G   6%
> /mnt/DP_disk3
> > > >
> > > > 10.1.0.123: /dev/sdb1             280G   66G  200G  25%
> /mnt/DP_disk1
> > > >
> > > > 10.1.0.123: /dev/sdc1             280G   65G  201G  25%
> /mnt/DP_disk2
> > > >
> > > > 10.1.0.123: /dev/sdd1             280G   65G  202G  25%
> /mnt/DP_disk3
> > > >
> > > > 10.1.0.126: /dev/sdb1             280G   14G  252G   6%
> /mnt/DP_disk1
> > > >
> > > > 10.1.0.126: /dev/sdc1             280G   14G  252G   6%
> /mnt/DP_disk2
> > > >
> > > > 10.1.0.126: /dev/sdd1             280G   13G  253G   5%
> /mnt/DP_disk3
> > > >
> > > > 2010/9/7 Tao Xie <xietao.mailbox@gmail.com>
> > > >
> > > > > I have a look at the following method in 0.89. Is the the following
> > > > line
> > > > > correct ?
> > > > >
> > > > > nRegions *= e.getValue().size();
> > > > >
> > > > >
> > > > > private int regionsToGiveOtherServers(final int
> numUnassignedRegions,
> > > > >     final HServerLoad thisServersLoad) {
> > > > >     SortedMap<HServerLoad, Set<String>> lightServers
=
> > > > >       new TreeMap<HServerLoad, Set<String>>();
> > > > >     this.master.getLightServers(thisServersLoad, lightServers);
> > > > >     // Examine the list of servers that are more lightly loaded
> than
> > > > this
> > > > > one.
> > > > >     // Pretend that we will assign regions to these more lightly
> > > > loaded
> > > > > servers
> > > > >     // until they reach load equal with ours. Then, see how many
> > > > regions
> > > > > are left
> > > > >     // unassigned. That is how many regions we should assign to
> this
> > > > > server.
> > > > >     int nRegions = 0;
> > > > >     for (Map.Entry<HServerLoad, Set<String>> e:
> > > > lightServers.entrySet()) {
> > > > >       HServerLoad lightLoad = new HServerLoad(e.getKey());
> > > > >       do {
> > > > >         lightLoad.setNumberOfRegions(lightLoad.getNumberOfRegions()
> +
> > > > 1);
> > > > >         nRegions += 1;
> > > > >       } while (lightLoad.compareTo(thisServersLoad) <= 0
> > > > >           && nRegions < numUnassignedRegions);
> > > > >       nRegions *= e.getValue().size();
> > > > >       if (nRegions >= numUnassignedRegions) {
> > > > >         break;
> > > > >       }
> > > > >     }
> > > > >     return nRegions;
> > > > >   }
> > > > >
> > > > >
> > > > >
> > > > > 2010/9/7 Jonathan Gray <jgray@facebook.com>
> > > > >
> > > > > That code does actually exist in the latest 0.89 release.
> > > > >>
> > > > >> It was a protection put in place to guard against a weird behavior
> > > > that we
> > > > >> had seen during load balancing.
> > > > >>
> > > > >> As Ryan suggests, this code was in need of a rewrite and was
just
> > > > >> committed last week to trunk/0.90.  If you're interested in the
> new
> > > > load
> > > > >> balancing code, it's in o.a.h.h.regionserver.LoadBalancer
> > > > >>
> > > > >> At the least, you should upgrade to 0.20.6 as there are some
> > > > important
> > > > >> fixes from 0.20.4 (until 0.90 is released, at which point everyone
> > > > should
> > > > >> move to it).
> > > > >>
> > > > >> JG
> > > > >>
> > > > >> > -----Original Message-----
> > > > >> > From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> > > > >> > Sent: Monday, September 06, 2010 7:10 PM
> > > > >> > To: user@hbase.apache.org
> > > > >> > Subject: Re: question about RegionManager
> > > > >> >
> > > > >> > That code was completely rewritten in 0.89/0.90... its pretty
> > > > dodgy so
> > > > >> > I'd
> > > > >> > highly consider upgrading to 0.89 asap.
> > > > >> > > hi, all
> > > > >> > >
> > > > >> > > I'm reading the code of RegionManager, I find in the
following
> > > > method
> > > > >> > there
> > > > >> > > is an situation when nRegionsToAssign <= nregions,
the code
> only
> > > > >> > assigns 1
> > > > >> > > region.
> > > > >> > > Is this correct? Hbase version 0.20.4.
> > > > >> > >
> > > > >> > > private void assignRegionsToMultipleServers(final HServerLoad
> > > > >> > > thisServersLoad,
> > > > >> > > final Set<RegionState> regionsToAssign, final
HServerInfo
> info,
> > > > >> > > final ArrayList<HMsg> returnMsgs) {
> > > > >> > > boolean isMetaAssign = false;
> > > > >> > > for (RegionState s : regionsToAssign) {
> > > > >> > > if (s.getRegionInfo().isMetaRegion())
> > > > >> > > isMetaAssign = true;
> > > > >> > > }
> > > > >> > > int nRegionsToAssign = regionsToAssign.size();
> > > > >> > > // Now many regions to assign this server.
> > > > >> > > int nregions = regionsPerServer(nRegionsToAssign,
> > > > thisServersLoad);
> > > > >> > > LOG.debug("Assigning for " + info + ": total nregions
to
> > > > assign=" +
> > > > >> > > nRegionsToAssign + ", nregions to reach balance=" +
nregions +
> > > > >> > > ", isMetaAssign=" + isMetaAssign);
> > > > >> > > if (nRegionsToAssign <= nregions) {
> > > > >> > > // I do not know whats supposed to happen in this case.
Assign
> > > > one.
> > > > >> > > LOG.debug("Assigning one region only (playing it safe..)");
> > > > >> > > assignRegions(regionsToAssign, 1, info, returnMsgs);
> > > > >> > > } else {
> > > > >>
> > > > >
> > > > >
> > >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message