Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B4529F7BD for ; Sun, 31 Mar 2013 05:56:34 +0000 (UTC) Received: (qmail 13673 invoked by uid 500); 31 Mar 2013 05:56:34 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 13648 invoked by uid 500); 31 Mar 2013 05:56:34 -0000 Mailing-List: contact user-help@helix.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.incubator.apache.org Delivered-To: mailing list user@helix.incubator.apache.org Received: (qmail 13621 invoked by uid 99); 31 Mar 2013 05:56:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 31 Mar 2013 05:56:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of g.kishore@gmail.com designates 74.125.82.54 as permitted sender) Received: from [74.125.82.54] (HELO mail-wg0-f54.google.com) (74.125.82.54) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 31 Mar 2013 05:56:26 +0000 Received: by mail-wg0-f54.google.com with SMTP id a12so1386832wgh.9 for ; Sat, 30 Mar 2013 22:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=k1jGeFaaJE8Llzd/8LL28/TcYbQ0SrW92kk4A4kGZMM=; b=FyeySo4cxuLvttIZvYanZDIXg6QH6EmcX6FHM2YD86kovPwItCLK0WYkTmIy/Uz5mh F9AFCtNL+3mqC4SsLJ2HZeIjGD4oBylVnnzMdDu2NUcR1axRzF5+p9zd97QfHNMClKu9 pQttGzYWladTThZFUnEnPzUyXa12kIYYS3Kpd4NlvIgsXj4Ctr+oSeSL7Ni6h9T8uAZM vC4vfMeWxy7MTtaU5qywoWGGYinp4Rgp93pyEN716UMv48qnhfJn32NFiL9q7D/WY153 jLTUWyfJ+0mNT6FYNZAer/RxNE4DZWUFFYLG2Rdsa/YD+PIRdHsjxaEdIPfgZ8EJG530 XHbg== MIME-Version: 1.0 X-Received: by 10.180.36.48 with SMTP id n16mr4767292wij.30.1364709365673; Sat, 30 Mar 2013 22:56:05 -0700 (PDT) Received: by 10.194.0.40 with HTTP; Sat, 30 Mar 2013 22:56:05 -0700 (PDT) In-Reply-To: References: Date: Sat, 30 Mar 2013 22:56:05 -0700 Message-ID: Subject: Re: Prevent failback to MASTER after failover From: kishore g To: user Content-Type: multipart/alternative; boundary=e89a8f502fa2f674f104d9322873 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f502fa2f674f104d9322873 Content-Type: text/plain; charset=ISO-8859-1 No you dont have to change the state model to achieve this. Intead of AUTO, AUTO_REBALANCE should work in your case. Simply change the ideal state to look like this. { "id": "Cluster", "simpleFields":{ "IDEAL_STATE_MODE":"AUTO_REBALANCE", "NUM_PARTITIONS": "1", "REPLICAS": "2", "STATE_MODEL_DEF_REF":"MasterSlave" }, "mapFields":{ }, "listFields":{ "Partition_0" : [ ] } } You can also achieve this in AUTO mode: when a node becomes master for a partition, as part of the transition change the preference list in the idealstate. So in this case change "Partition_0" : [ "node_1", "node_2" ] to "Partition_0" : [ "node_2", "node_1" ] when node_2 becomes master. In the next release, you will be able to add custom rebalancer code which will allow you to make this change in the controller easily. thanks, Kishore G On Sat, Mar 30, 2013 at 10:21 PM, Ming Fang wrote: > Hi Kishore > > Our system requires deterministic placement of the MASTER and SLAVE. > This is a sample of the idealstate file we're using > > { > "id": "Cluster", > "simpleFields":{ > "IDEAL_STATE_MODE":"AUTO", > "NUM_PARTITIONS": "1", > "REPLICAS": "2", > "STATE_MODEL_DEF_REF":"MasterSlave" > }, > "mapFields":{ > }, > "listFields":{ > "Partition_0" : [ "node_1", "node_2" ] > } > } > > > In this example, node_1 is the MASTER. > If node_1 dies then node_2 will take over. > But if node_1 then get restarted, it will try to become MASTER again. > We normally keep the died node down to avoid this problem. > But I was hoping for a more elegant solution. > > One solution would be for node_1 to come up and realizes that node_2 has > taken over due to the previous failure. > In that case node_1 will decide to remain as a SLAVE node instead. > Should this be done by the Controller instead? > Should I create a new statemodel other than MASTER/SLAVE? > > On Mar 31, 2013, at 12:50 AM, kishore g wrote: > > Hi MIng, > > There are couple of ways you can achieve that. Before providing an answer, > how many partitions do you have. Did you generate the idealstate yourself > or used Helix to come up with initial idealstate? > > The reason old master tries to become a master again is to distribute the > load among the nodes currently alive. Otherwise the old node that comes > back will never become a master for any partition and will remain idle > until another failure happens in the system. > > thanks, > Kishore G > > > On Sat, Mar 30, 2013 at 8:01 PM, Ming Fang wrote: > >> We're using MASTER SLAVE in AUTO model. >> When the MASTER is killed, the failover is working properly as the SLAVE >> transitions to become MASTER. >> However if the failed MASTER is restarted, it will try to become MASTER >> again. >> This is causing a problem in our business logic. >> Is there a way to prevent the failed instance from becoming MASTER again? > > > > --e89a8f502fa2f674f104d9322873 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
No you dont have to change the state model to a= chieve this.

Intead of AUTO, AUTO_REBALANCE shoul= d work in your case. Simply change the ideal state to look like this.

{
=A0=A0=A0=A0"id": "Cluster",


=A0=A0=A0=A0<=
span style=3D"margin:0px;padding:0px;border:0px;color:rgb(0,0,128)">"s=
impleFields":=
{


=A0=A0=A0=A0=
=A0=A0=A0=A0"IDEAL_STATE_MODE":"AUTO_REBALANCE",


=A0=A0=A0=A0=
=A0=A0=A0=A0"NUM_PARTITIONS": "1",


=A0=A0=A0=A0=
=A0=A0=A0=A0"REPLICAS": "2",


=A0=A0=A0=A0=
=A0=A0=A0=A0"STATE_MODEL_DEF_REF":"MasterSlave"


=A0=A0=A0=A0<=
span style=3D"margin:0px;padding:0px;border:0px">},
=A0=A0=A0=A0"mapFields&q=
uot;:{

=A0=A0=A0=A0<=
span style=3D"margin:0px;padding:0px;border:0px">},
=A0=A0=A0=A0"listFields&=
quot;:{

=A0=A0=A0=A0=
=A0=A0=A0=A0"Partition_0" : [ ]


=A0=A0=A0=A0<=
span style=3D"margin:0px;padding:0px;border:0px">}
}
You can also achieve this in AUTO mode: when a node becomes mast= er for a partition, as part of the transition change the preference list in= the idealstate. So in this case change=A0"Partition_0" : [= "node_1", "node_2" ] to &qu= ot;Partition_0" : [ "node_2", "node_1" ] when node_2 becomes master.

In the next release, you will be able= to add custom rebalancer code which will allow you to make this change in = the controller easily.

thanks,
Kishore G



On Sat, Mar 30, 2013 at 10:21 PM, Ming Fang <m= ingfang@mac.com> wrote:
Hi Kisho= re

Our system requires deterministic placement of the MA= STER and SLAVE.
This is a sample of the idealstate file we're using

= 
{
=A0=A0=A0=A0"id": "Cluster",


=A0=A0=A0=A0<=
span style=3D"margin:0px;padding:0px;border:0px;color:rgb(0,0,128)">"s=
impleFields":=
{


=A0=A0=A0=A0=
=A0=A0=A0=A0"IDEAL_STATE_MODE":"AUTO",


=A0=A0=A0=A0=
=A0=A0=A0=A0"NUM_PARTITIONS": "1",


=A0=A0=A0=A0=
=A0=A0=A0=A0"REPLICAS": "2",


=A0=A0=A0=A0=
=A0=A0=A0=A0"STATE_MODEL_DEF_REF":"MasterSlave"


=A0=A0=A0=A0<=
span style=3D"margin:0px;padding:0px;border:0px">},
=A0=A0=A0=A0"mapFields&q=
uot;:{

=A0=A0=A0=A0<=
span style=3D"margin:0px;padding:0px;border:0px">},
=A0=A0=A0=A0"listFields&=
quot;:{

=A0=A0=A0=A0=
=A0=A0=A0=A0"Partition_0" : [ "node_1", "node_2" ]


=A0=A0=A0=A0<=
span style=3D"margin:0px;padding:0px;border:0px">}
}

In this example, node_1 is the MASTER.
= If node_1 dies then node_2 will take over.
But if node_1 then get= restarted, it will try to become MASTER again.
We normally keep = the died node down to avoid this problem.
But I was hoping for a more elegant solution.

One solution would be for node_1 to come up and realizes that node_2 has t= aken over due to the previous failure.
In that case node_1 will d= ecide to remain as a SLAVE node instead.
Should this be done by the Controller instead?
Should I crea= te a new statemodel other than MASTER/SLAVE?

On Mar 31, 2013, at 12:50 AM, kishore g <g.kishore@gmail.com&g= t; wrote:

Hi MIng,

= There are couple of ways you can achieve that. Before providing an answer, = how many partitions do you have. Did you generate the idealstate yourself o= r used Helix to come up with initial idealstate?

The reason old master tries to become a master again is= to distribute the load among the nodes currently alive. Otherwise the old = node that comes back will never become a master for any partition and will = remain idle until another failure happens in the system.

thanks,
Kishore G


On Sat, Mar 30, 2013 at 8:01 = PM, Ming Fang <mingfang@mac.com> wrote:
We're using MASTER SLAVE in AUTO model.<= br> When the MASTER is killed, the failover is working properly as the SLAVE tr= ansitions to become MASTER.
However if the failed MASTER is restarted, it will try to become MASTER aga= in.
This is causing a problem in our business logic.
Is there a way to prevent the failed instance from becoming MASTER again?



--e89a8f502fa2f674f104d9322873--