Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BEB241031F for ; Mon, 20 Jan 2014 06:00:45 +0000 (UTC) Received: (qmail 91854 invoked by uid 500); 20 Jan 2014 06:00:45 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 91765 invoked by uid 500); 20 Jan 2014 06:00:41 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 90093 invoked by uid 99); 20 Jan 2014 06:00:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jan 2014 06:00:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of g.kishore@gmail.com designates 209.85.212.171 as permitted sender) Received: from [209.85.212.171] (HELO mail-wi0-f171.google.com) (209.85.212.171) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jan 2014 06:00:32 +0000 Received: by mail-wi0-f171.google.com with SMTP id cc10so2767227wib.4 for ; Sun, 19 Jan 2014 22:00:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=fWRmkMCanb/Vck3xBQWK6OcooAx+xRD5i8lgvRhz06A=; b=x4+ZV0wpVxzFu+9YWhC2+YAC4UAzzS1SH6f3XlyUtgQBhxX3GqkIq0EI8c4G6oDElY Q2s87xvJUQHB6CZkwDH6NB8JHvtAftqrffsy43wD6++W6B/U/8LtUYREcai8IaI2aQIP JGY0aBX7G5AJUvVFyqyvukXa1UsTnHdPQ39A0gMiLbz/O75F4O63vkvHAMdNmvGkDDWn TiKV9Oae5jmUmk5Gebp60NOsAi7jL8xLj4QOTZgO8zzPBC0crbU5hKWDyw9uXZMIop5S bjBY/d1DjF19718IgGhR0HY97J4xCXunAeFDtse02j+WFoVK0U7/NeM8wqKvjKK6G81Z Lxcg== MIME-Version: 1.0 X-Received: by 10.180.211.141 with SMTP id nc13mr12358491wic.1.1390197611667; Sun, 19 Jan 2014 22:00:11 -0800 (PST) Received: by 10.194.78.143 with HTTP; Sun, 19 Jan 2014 22:00:11 -0800 (PST) In-Reply-To: References: Date: Sun, 19 Jan 2014 22:00:11 -0800 Message-ID: Subject: Re: TaskRebalancer From: kishore g To: user@helix.apache.org Content-Type: multipart/alternative; boundary=001a11c3709ccfa61004f0609ae4 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c3709ccfa61004f0609ae4 Content-Type: text/plain; charset=ISO-8859-1 Actually that makes a lot of sense. Let me look at that. On Sun, Jan 19, 2014 at 8:49 PM, Kanak Biscuitwala wrote: > > This sounds a lot like what we did in AutoRebalanceStrategy. There's an > interface called ReplicaPlacementScheme that the algorithm calls into, and > a DefaultPlacementScheme that just does evenly balanced assignment. > > The simplest thing we could do is have a task rebalancer config and set a > switch for which placement scheme to use. The current task rebalancer > already has to specify things like the DAG, so this could just be another > field to add on. > > > Date: Sun, 19 Jan 2014 13:14:33 -0800 > > Subject: Re: TaskRebalancer > > From: g.kishore@gmail.com > > To: dev@helix.apache.org > > CC: dev@helix.incubator.apache.org; user@helix.incubator.apache.org > > > > > Thanks Jason, I was looking at the rebalancer. Looks like target resource > > is mandatory. What do you suggest is the right way to make target > resource > > optional. > > > > This is my understanding of what task rebalancer is doing today. > > > > It assumes that the system is already hosting a resource something like a > > database, index etc. Now one can use the task framework to launch > arbitrary > > tasks on nodes hosting these resources. For example lets say there is a > > database MyDB with 3 partitions and 2 replicas and using Master Slave > state > > model and 3 nodes N1 N2 N3. In a happy state the cluster might look like > > this > > > > { > > "id":"MyDB", > > "mapFields":{ > > "MyDB_0":{ > > "N1":"MASTER", > > "N2":"SLAVE" > > }, > > "MyDB_1":{ > > "N2":"MASTER", > > "N3":"SLAVE" > > }, > > "MyDB_2":{ > > "N1":"SLAVE", > > "N3":"MASTER" > > } > > } > > } > > > > Lets say one wants to take backup of these databases but run only the > > SLAVEs. One can define the back up task and launch 3 back up tasks (one > for > > each partition) only on SLAVEs. > > > > What we have currently works perfectly for this scenario. One has to > simply > > define the target resource and state for the backup tasks and they will > be > > launched in appropriate place. So in this scenario, back task for > > partitions 0,1,2 will be launched at N2, N3, and N1. > > > > But what if the tasks dont have any target resource and can be run on any > > node N1 N2 or N3 and the only requirement is distribute the tasks evenly. > > > > We should decouple the logic of where a task is placed from the logic of > > distributing the tasks. For example, we can abstract out the placement > > constraint from the rebalancer logic. So we can have a placement provider > > that computes placement randomly and one that computes placement based on > > another resource. Probably another one that computes placement based on > > data locality. > > > > What is the right way to approach this ? > > > > thanks, > > Kishore G > > > > > > On Sun, Jan 19, 2014 at 10:12 AM, Zhen Zhang > wrote: > > > > > TestTaskRebalancer and TestTaskRebalancerStopResume are examples. > > > > > > Thanks, > > > Jason > > > > > > > > > On Sun, Jan 19, 2014 at 9:20 AM, kishore g > wrote: > > > > > > > Hi, > > > > > > > > I am trying to use TaskRebalancer but not able to understand how it > > > works, > > > > is there any example I can try? > > > > > > > > thanks, > > > > Kishore G > > > > > > > > --001a11c3709ccfa61004f0609ae4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Actually that makes a lot of sense. Let me look at that. <= br>


On S= un, Jan 19, 2014 at 8:49 PM, Kanak Biscuitwala <kanak.b@hotmail.com&= gt; wrote:

This sounds a lot like what we did in AutoRebalan= ceStrategy. There's an interface called ReplicaPlacementScheme that the= algorithm calls into, and a DefaultPlacementScheme that just does evenly b= alanced assignment.

The simplest thing we could do is have a task rebalancer con= fig and set a switch for which placement scheme to use. The current task re= balancer already has to specify things like the DAG, so this could just be = another field to add on.

> Date: Sun, 19 Jan 2014 13:14:33 -0800
> Subj= ect: Re: TaskRebalancer
> From: g.kishore@gmail.com
> To: dev@helix.apache.org
> CC: dev@helix.incubator.apache.org; user@helix.incubator.apache.org

>
> Thanks Jason, I was looking at the rebalancer. Looks like= target resource
> is mandatory. What do you suggest is the right way= to make target resource
> optional.
>
> This is my unde= rstanding of what task rebalancer is doing today.
>
> It assumes that the system is already hosting a resource some= thing like a
> database, index etc. Now one can use the task framewor= k to launch arbitrary
> tasks on nodes hosting these resources. For e= xample lets say there is a
> database MyDB with 3 partitions and 2 replicas and using Master Slave = state
> model and 3 nodes N1 N2 N3. In a happy state the cluster migh= t look like
> this
>
> {
> "id":"= MyDB",
> "mapFields":{
> "MyDB_0":{
> = "N1":"MASTER",
> "N2":"SL= AVE"
> },
> "MyDB_1":{
> &qu= ot;N2":"MASTER",
> "N3":"SLAVE"
> },
> &quo= t;MyDB_2":{
> "N1":"SLAVE",
> = "N3":"MASTER"
> }
> }
> } >
> Lets say one wants to take backup of these databases but run = only the
> SLAVEs. One can define the back up task and launch 3 back = up tasks (one for
> each partition) only on SLAVEs.
>
> = What we have currently works perfectly for this scenario. One has to simply=
> define the target resource and state for the backup tasks and they wil= l be
> launched in appropriate place. So in this scenario, back task = for
> partitions 0,1,2 will be launched at N2, N3, and N1.
> > But what if the tasks dont have any target resource and can be run on = any
> node N1 N2 or N3 and the only requirement is distribute the tas= ks evenly.
>
> We should decouple the logic of where a task is= placed from the logic of
> distributing the tasks. For example, we can abstract out the placement=
> constraint from the rebalancer logic. So we can have a placement p= rovider
> that computes placement randomly and one that computes plac= ement based on
> another resource. Probably another one that computes placement based o= n
> data locality.
>
> What is the right way to approach= this ?
>
> thanks,
> Kishore G
>
>
> On Sun, Jan 19, 2014 at 10:12 AM, Zhen Zhang <nehzgnahz@gmail.com> wrote:
&= gt;
> > TestTaskRebalancer and TestTaskRebalancerStopResume are e= xamples.
> >
> > Thanks,
> > Jason
> >
> >=
> > On Sun, Jan 19, 2014 at 9:20 AM, kishore g <g.kishore@gmail.com> wrot= e:
> >
> > > Hi,
> > >
> > > I am tr= ying to use TaskRebalancer but not able to understand how it
> > w= orks,
> > > is there any example I can try?
> > > > > > thanks,
> > > Kishore G
> > >
>= ; >

--001a11c3709ccfa61004f0609ae4--