hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Dyer <rd...@iastate.edu>
Subject Re: Heterogeneous cluster
Date Sat, 08 Dec 2012 23:50:21 GMT
I of course can not speak for Jean-Marc, however my use case is not very
corporate.  It is a small cluster (9 nodes) and only 1 of those nodes is
different (drastically different).

And yes, I configured it so that node has a lot more map slots.  However,
the problem is HBase balances without regard to that and thus even though
more map tasks run on those nodes they are not data-local!  If I have a
balancer that is able to keep more regions on that particular node, then
the data locality of my map tasks is improved.


On Sat, Dec 8, 2012 at 5:45 PM, Michael Segel <michael_segel@hotmail.com>wrote:

> Take what I say with a grain of kosher salt. (Its what they put on your
> drink glasses because the grains are bigger. ;-)
>
> I think what you are doing is cool hack, however in the bigger picture,
> you shouldn't have to do this with your load balancer. Also it doesn't
> matter if you think about ti.
>
> With a heterogenous cluster, you will not share the same configuration
> across all machines in the cluster. You will change the number of slots per
> node based on its capacity.
> That will limit what amount of work could be done on the same cluster.
>
> You could also consider playing with the rack aware aspects of your
> cluster.
> You could make all of your 2CPU machines in the same rack.
>
> In theory... machine, rack , second rack is how the data is distributed.
> In theory if the 2CPU cores are neighbors, then the 2nd and or 3rd copy
> goes to another machine.
>
> Trying to write a custom balancer, may be a good hack, but not good in
> terms of corporate life.
>
> Just saying!
>
> -Mike
>
> On Dec 8, 2012, at 1:34 PM, Jean-Marc Spaggiari <jean-marc@spaggiari.org>
> wrote:
>
> > Hi,
> >
> > It's not yet available anywhere. I will post it today or tomorrow,
> > just the time to remove some hardcoding I did into it ;) It's a quick
> > and dirty PerformanceBalancer. It's not a CPULoadBalencer.
> >
> > Anyway, I will give more details over the week-end, but there is
> > absolutly nothing extraordinaire with it.
> >
> > JM
> >
> > 2012/12/8, Robert Dyer <rdyer@iastate.edu>:
> >> I too am interested in this custom load balancer, as I was actually just
> >> starting to look into writing one that does the same thing for
> >> my heterogeneous cluster!
> >>
> >> Is this available somewhere?
> >>
> >> On Sat, Dec 8, 2012 at 9:17 AM, James Chang <james.bigdata@gmail.com>
> >> wrote:
> >>
> >>>     By the way, I saw you mentioned that you
> >>> have built a "LoadBalancer", could you kindly
> >>> share some detailed info about it?
> >>>
> >>> Jean-Marc Spaggiari 於 2012年12月8日星期六寫道:
> >>>
> >>>> Hi,
> >>>>
> >>>> Here is the situation.
> >>>>
> >>>> I have an heterogeneous cluster with 2 cores CPUs, 4 cores CPUs and
8
> >>>> cores CPUs servers. The performances of those different servers allow
> >>>> them to handle different size of load. So far, I built a LoadBalancer
> >>>> which balance the regions over those servers based on the
> >>>> performances. And it’s working quite well. The RowCounter went down
> >>>> from 11 minutes to 6 minutes. However, I can still see that the tasks
> >>>> are run on some servers accessing data on other servers, which
> >>>> overwhelme the bandwidth and slow done the process since some 2 cores
> >>>> servers are assigned to count some rows hosted on 8 cores servers.
> >>>>
> >>>> I’m looking for a way to “force” the tasks to run on the servers
where
> >>>> the regions are assigned.
> >>>>
> >>>> I first tried to reject the tasks on the Mapper setup method when the
> >>>> data was not local to see if the tracker will assign it to another
> >>>> server. No. It’s just failing and mostly not re-assigned. I tried
> >>>> IOExceptions, RuntimeExceptions, InterruptionExceptions with no
> >>>> success.
> >>>>
> >>>> So now I have 3 possible options.
> >>>>
> >>>> The first one is to move from the MapReduce to the Coprocessor
> >>>> EndPoint. Running locally on the RegionServer, it’s accessing only
the
> >>>> local data and I can manually reject all what is not local. Therefor
> >>>> it’s achieving my needs, but it’s not my preferred options since
I
> >>>> would like to keep the MR features.
> >>>>
> >>>> The second option is to tell Hadoop where the tasks should be
> >>>> assigned. Should that be done by HBase? By Hadoop? I don’t know.
> >>>> Where? I don’t know either. I have started to look at JobTracker and
> >>>> JobInProgress code but it seems it will be a big task. Also, doing
> >>>> that will mean I will have to re-patch the distributed code each time
> >>>> I’m upgrading the version, and I will have to redo everything when
I
> >>>> will move from 1.0.x to 2.x…
> >>>>
> >>>> Third option is to not process the task if the data is not local. I
> >>>> mean, on the map method, simply have a if (!local) return; right from
> >>>> the beginning and just do nothing. This will not work for things like
> >>>> RowCount since all the entries are required, but for some of my
> >>>> usecases this might work where I don’t necessary need all the data
to
> >>>> be processed. I will not be efficient stlil the task will still scan
> >>>> the entire region.
> >>>>
> >>>> My preferred option is definitively the 2nd one, but it seems also to
> >>>> be the most difficult one. The Third one is very easy to implement.
> >>>> Need 2 lines to see if the data is local. But it’s not working for
all
> >>>> the scenarios, and is more like a dirty fix. The coprocessor option
> >>>> might be doable too since I already have all the code for my MapReduce
> >>>> jobs. So it might be an acceptable option.
> >>>>
> >>>> I’m wondering if anyone already faced this situation and worked on
> >>>> something, and if not, do you have any other ideas/options to propose,
> >>>> or can someone point me to the right classes to look at to implement
> >>>> the solution 2?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> JM
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Robert Dyer
> >> rdyer@iastate.edu
> >>
> >
>
>


-- 

Robert Dyer
rdyer@iastate.edu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message