Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DAAC611CED for ; Wed, 9 Apr 2014 20:40:34 +0000 (UTC) Received: (qmail 38661 invoked by uid 500); 9 Apr 2014 20:40:34 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 38569 invoked by uid 500); 9 Apr 2014 20:40:33 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 38558 invoked by uid 99); 9 Apr 2014 20:40:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 20:40:33 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of kanak.b@hotmail.com designates 65.54.190.35 as permitted sender) Received: from [65.54.190.35] (HELO bay0-omc1-s24.bay0.hotmail.com) (65.54.190.35) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 20:40:26 +0000 Received: from BAY182-W24 ([65.54.190.60]) by bay0-omc1-s24.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 9 Apr 2014 13:40:05 -0700 X-TMN: [wjML7xOVwuMEenWG/trFs50i7WZ+mb3z] X-Originating-Email: [kanak.b@hotmail.com] Message-ID: From: Kanak Biscuitwala To: "user@helix.apache.org" Subject: RE: keeping the master node up during bootstrap Date: Wed, 9 Apr 2014 13:40:05 -0700 Importance: Normal In-Reply-To: References: ,,,,<61F5E811-432F-4E18-B4AE-C49C04EC0123@gmail.com>,,,,, Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 09 Apr 2014 20:40:05.0698 (UTC) FILETIME=[E3AEC220:01CF5433] X-Virus-Checked: Checked by ClamAV on apache.org =0A= Based on the result of the conversation=2C we found the following:=0A= =0A= 1. 0.6.x doesn't support partition constraints. Created=9Ahttps://issues.ap= ache.org/jira/browse/HELIX-426=0A= 2. 0.7.x doesn't honor partition constraints correctly. Created=9Ahttps://i= ssues.apache.org/jira/browse/HELIX-425=0A= =0A= We will try to fix these tomorrow.=0A= =0A= Kanak=0A= ________________________________=0A= > Date: Wed=2C 9 Apr 2014 12:51:10 -0700 =0A= > Subject: Re: keeping the master node up during bootstrap =0A= > From: vlad.gm@gmail.com =0A= > To: user@helix.apache.org =0A= > =0A= > Sure! I'll join the channel! =0A= > =0A= > =0A= > On Wed=2C Apr 9=2C 2014 at 12:41 PM=2C kishore g =0A= > > wrote: =0A= > Hi Vlad=2C =0A= > =0A= > I have some questions. Can you join the IRC channel #apachehelix. =0A= > =0A= > thanks=2C =0A= > Kishore G =0A= > =0A= > =0A= > On Wed=2C Apr 9=2C 2014 at 11:35 AM=2C =0A= > vlad.gm@gmail.com =0A= > > wrote: =0A= > Upon some further testing=2C it seems that the controller does not =0A= > execute the events in the right sequence. =0A= > =0A= > Here are the results of some of my testing. Assume that we have a =0A= > partition NEWPROFILE_5 with the ideal state: =0A= > =0A= > "NEWPROFILE_5" : { =0A= > =0A= > "pf1.apps-pf.dev.docker_12000" : "SLAVE"=2C =0A= > =0A= > "pf2.apps-pf.dev.docker_12000" : "MASTER" =0A= > =0A= > } =0A= > =0A= > I boot the host pf1 and a few minutes later the host pf2. In the =0A= > controller logs I see=2C when doing a grep for NEWPROFILE_5: =0A= > =0A= > 2014-04-08 17:04:35=2C309 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message 69b4eddf-ac5f-4726-9d6b-bac742ad082e to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE =0A= > to:MASTER =0A= > =0A= > 2014-04-08 17:27:08=2C187 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message a221b1ac-0807-425e-9062-6507e45b0bfb to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:OFFLINE =0A= > to:BOOTSTRAP =0A= > =0A= > 2014-04-08 17:27:10=2C164 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message 73ed85fd-49c9-46a5-b262-687d612c7d06 to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:BOOTSTRAP =0A= > to:SLAVE =0A= > =0A= > 2014-04-08 17:27:11=2C868 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message fb21aecc-68cf-4b9f-9718-aa6ed535c29d to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE =0A= > to:MASTER =0A= > =0A= > 2014-04-08 17:28:22=2C978 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message ea441d18-b1f3-4ceb-96a2-3262cab1dfbe to =0A= > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:OFFLINE =0A= > to:BOOTSTRAP =0A= > =0A= > 2014-04-08 17:28:22=2C978 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message f36b4d64-c790-413b-b9fa-915b9539d28c to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:MASTER =0A= > to:SLAVE =0A= > =0A= > 2014-04-08 17:28:26=2C065 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message 201429e1-e810-4017-b3ef-fb5930ac2192 to =0A= > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:BOOTSTRAP =0A= > to:SLAVE =0A= > =0A= > 2014-04-08 17:28:28=2C238 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message 4a1fb64c-1063-4e49-a995-946d2dd25733 to =0A= > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE =0A= > to:MASTER =0A= > =0A= > That is=2C the controller issues an offline->bootstrap command to pf-2=2C= =0A= > but then issues a master->slave command to of-1 before bringing pf-2 up = =0A= > as a slave as well (the last step before promotion to master). Since =0A= > the bootstrap->slave that follows takes time=2C the system spends time = =0A= > without a master for the partition. =0A= > =0A= > The state model definition was: =0A= > public static StateModelDefinition defineStateModel() { =0A= > StateModelDefinition.Builder builder =3D =0A= > new StateModelDefinition.Builder(KVHelixDefinitions.STATE_MODEL_NAME)=3B = =0A= > // Add states and their rank to indicate priority. Lower the rank higher = the =0A= > // priority =0A= > builder.addState(MASTER=2C 1)=3B =0A= > builder.addState(SLAVE=2C 2)=3B =0A= > builder.addState(BOOTSTRAP=2C 3)=3B =0A= > builder.addState(OFFLINE)=3B =0A= > builder.addState(DROPPED)=3B =0A= > // Set the initial state when the node starts =0A= > builder.initialState(OFFLINE)=3B =0A= > =0A= > // Add transitions between the states. =0A= > builder.addTransition(OFFLINE=2C BOOTSTRAP=2C 3)=3B =0A= > builder.addTransition(BOOTSTRAP=2C SLAVE=2C 2)=3B =0A= > builder.addTransition(SLAVE=2C MASTER=2C 1)=3B =0A= > builder.addTransition(MASTER=2C SLAVE=2C 4)=3B =0A= > builder.addTransition(SLAVE=2C OFFLINE=2C 5)=3B =0A= > builder.addTransition(OFFLINE=2C DROPPED=2C 6)=3B =0A= > =0A= > // set constraints on states. =0A= > // static constraint =0A= > builder.upperBound(MASTER=2C 1)=3B =0A= > // dynamic constraint=2C R means it should be derived based on the replic= ation =0A= > // factor. =0A= > builder.dynamicUpperBound(SLAVE=2C "R")=3B =0A= > =0A= > StateModelDefinition statemodelDefinition =3D builder.build()=3B =0A= > =0A= > assert(statemodelDefinition.isValid())=3B =0A= > =0A= > return statemodelDefinition=3B =0A= > } =0A= > =0A= > I have tried reversing the values of the transition priorities. In this = =0A= > case=2C the controller log file looked as follows: =0A= > =0A= > 2014-04-09 11:17:52=2C831 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message 2b29a319-c1c6-4042-b1ad-3e3c1b5092a7 to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:OFFLINE =0A= > to:BOOTSTRAP =0A= > =0A= > 2014-04-09 11:17:55=2C672 (Thread-2) MessageGenerationStage INFO: Message= =0A= > hasn't been removed for pf1.apps-pf.dev.docker_12000 to =0A= > transitNEWPROFILE_5 to BOOTSTRAP=2C desiredState: MASTER =0A= > =0A= > 2014-04-09 11:17:57=2C047 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message b1ca701d-65f1-46b9-9ae4-286400d6d266 to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:BOOTSTRAP =0A= > to:SLAVE =0A= > =0A= > 2014-04-09 11:17:58=2C888 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message fe10228f-8f5b-4133-964a-5f6c7e60b0e6 to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE =0A= > to:MASTER =0A= > =0A= > 2014-04-09 11:23:26=2C117 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message 6252a4e6-0ab8-490a-a51d-c47195c434b5 to =0A= > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:MASTER =0A= > to:SLAVE =0A= > =0A= > 2014-04-09 11:23:26=2C117 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message 18bbf028-cb51-4162-8226-a6564a121986 to =0A= > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:OFFLINE =0A= > to:BOOTSTRAP =0A= > =0A= > 2014-04-09 11:23:33=2C462 (Thread-2) MessageGenerationStage INFO: Message= =0A= > hasn't been removed for pf2.apps-pf.dev.docker_12000 to =0A= > transitNEWPROFILE_5 to BOOTSTRAP=2C desiredState: MASTER =0A= > =0A= > 2014-04-09 11:23:33=2C892 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message c7fc4983-9d71-4dc4-bfee-2ad69e4de411 to =0A= > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:BOOTSTRAP =0A= > to:SLAVE =0A= > =0A= > 2014-04-09 11:23:35=2C933 (Thread-2) TaskAssignmentStage INFO: Sending = =0A= > Message 75e715ed-3d53-4e39-b1e7-44695e4bfa03 to =0A= > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE =0A= > to:MASTER =0A= > =0A= > That is=2C the transition for master->slave for pf1 was executed before = =0A= > taking any action on pf2=2C clearly the opposite of the right order. =0A= > =0A= > =0A= > On Tue=2C Apr 8=2C 2014 at 2:19 PM=2C Kanak Biscuitwala =0A= > > wrote: =0A= > =0A= > Looks good=2C thanks for sharing! =0A= > =0A= > Kanak =0A= > ________________________________ =0A= >> Date: Tue=2C 8 Apr 2014 14:08:28 -0700 =0A= >> Subject: Re: keeping the master node up during bootstrap =0A= >> From: vlad.gm@gmail.com =0A= >> To: user@helix.apache.org =0A= >> =0A= >> My modified code looks like: =0A= >> =0A= >> /* Setup a Helix cluster for the KVStore */ =0A= >> public static void setupCluster() { =0A= >> assert(cluster !=3D null)=3B =0A= >> clusterSetup.addCluster(cluster=2C true)=3B =0A= >> =0A= >> =C1 =C1 =C1 =C1 ConstraintItemBuilder constraintItemBuilder =3D new =0A= >> ConstraintItemBuilder()=3B =0A= >> =0A= >> =C1 =C1 =C1 =C1 constraintItemBuilder =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> .addConstraintAttribute(ConstraintAttribute.MESSAGE_TYPE.toString()=2C = =0A= >> "STATE_TRANSITION") =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> .addConstraintAttribute(ConstraintAttribute.PARTITION.toString()=2C ".*"= ) =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> .addConstraintAttribute(ConstraintAttribute.CONSTRAINT_VALUE.toString()= =2C =0A= >> "1")=3B =0A= >> =0A= >> =C1 =C1 =C1 =C1 clusterSetup.getClusterManagementTool().setConstraint(cl= uster=2C =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 ClusterConstraints.ConstraintType.MESSAG= E_CONSTRAINT=2C =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 "constraint1"=2C constraintItemBuilder.b= uild())=3B =0A= >> =C1 =C1 } =0A= >> =0A= >> I will try to see whether it works in every situation. =0A= >> =0A= >> Regards=2C =0A= >> Vlad =0A= >> =0A= >> =0A= >> On Tue=2C Apr 8=2C 2014 at 8:59 AM=2C Vlad Balan =0A= >> =0A= > >> =0A= > wrote: =0A= >> Hi Kishore=2C =0A= >> =0A= >> I managed to implement the bootstrapping using the constraint and it =0A= >> appears to be running as expected. I will post my code shortly. =0A= >> =0A= >> Regards=2C =0A= >> Vlad =0A= >> =0A= >> On Apr 8=2C 2014=2C at 8:27 AM=2C kishore g =0A= >> =0A= > >> =0A= > wrote: =0A= >> =0A= >> Hi Vlad=2C =0A= >> =0A= >> Did you get a chance to play with the constraint.=C1 I can write a sampl= e =0A= >> code today to try this. =0A= >> =0A= >> Thanks=2C =0A= >> Kishore G =0A= >> =0A= >> =0A= >> On Thu=2C Apr 3=2C 2014 at 5:45 PM=2C =0A= >> =0A= > vlad.gm@gmail.com> =0A= >> =0A= > >> =0A= > wrote: =0A= >> =0A= >> Thank you Kanak and Kishore! I will try enforcing the per-partition =0A= >> constraint and let you know if somehow it does not work. I was looking = =0A= >> at the throttling documentation=2C but somehow missed that a =0A= >> per-partition constraint was an option! =0A= >> =0A= >> Regards=2C =0A= >> Vlad =0A= >> =0A= >> =0A= >> On Thu=2C Apr 3=2C 2014 at 5:42 PM=2C kishore g =0A= >> =0A= > >> =0A= > wrote: =0A= >> Hi Vlad=2C =0A= >> =0A= >> You can try setting the transition priority order and a constraint that = =0A= >> there should be only one transition per partition across the cluster. = =0A= >> =0A= >> So the transition priority could be something like =0A= >> =0A= >> Slave-Master =0A= >> Offfline -> Bootstrap =0A= >> Bootstrap->Slave =0A= >> Slave->Master =0A= >> =0A= >> For the rest not sure if order matters. =0A= >> =0A= >> Also set the max transitions constraint to 1 per partition. =0A= >> =0A= >> The reason I put Slave-Master before Offline->Bootstrap is to ensure =0A= >> that availability is given more importance. For example if you have 3 = =0A= >> nodes=2C N1=2C N2=2C N3. N1 is Master=2C N2 is Slave=2C and N3 is down. = If N1 =0A= >> goes down and N3 comes up at the same time. We probably dont want to =0A= >> wait for N3 to bootstrap before promoting N2 to Master. =0A= >> =0A= >> I haven't tested this but assuming the constraints enforcement works=2C = =0A= >> this should do the trick. =0A= >> =0A= >> Does this make sense? Let me know if this does not work=2C we can add a = =0A= >> test case. =0A= >> =0A= >> thanks=2C =0A= >> Kishore G =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> =0A= >> On Thu=2C Apr 3=2C 2014 at 4:57 PM=2C =0A= >> =0A= > vlad.gm@gmail.com> =0A= >> =0A= > >> =0A= > wrote: =0A= >> =0A= >> Dear all=2C =0A= >> =0A= >> I am trying to construct a state model with the following transition =0A= > diagram: =0A= >> =0A= >> OFFLINE -> BOOTSTRAPPING <---> SLAVE <-----> MASTER =0A= >> =C1 =C1 =C1 =C1 =C1<----------------------------------- =0A= >> =0A= >> That is=2C an offline mode can go into a bootstraping state=2C from the = =0A= >> bootstrap state it can go into a slave state=2C =0A= >> from slave it can go from master=2C from master to slave and from slave = =0A= >> it can go offline. =0A= >> =0A= >> Assume that if I have a partition with two nodes pf1 and pf2 and a =0A= >> partition partition_0 with the following ideal state: =0A= >> =0A= >> partition_0: pf2: MASTER pf1: SLAVE=2C =0A= >> =0A= >> and that currently pf1 is serving as a master. When pf2 boots=2C Helix = =0A= >> will issue=2C almost simultaneously=2C two commands: =0A= >> for pf1: transition from MASTER to SLAVE =0A= >> for pf2: transition from BOOTSTRAPPING to SLAVE =0A= >> =0A= >> My understanding is that this happens since Helix is trying to execute = =0A= >> as many commands in parallel and since the last state =0A= >> has pf2 as master. However=2C the transition from BOOTSTRAPPING to SLAVE= =0A= >> for pf2 involves a long data copy step=2C so =0A= >> I would like to keep pf1 as a master in the meanwhile. I tried =0A= >> prioritizing the transition from BOOTSTRAPPING to SLAVE =0A= >> over the transition from MASTER to SLAVE=2C however Helix still issues = =0A= >> them in parallel (as it should). =0A= >> =0A= >> I was wondering what my options would be in order to keep the master up = =0A= >> while the future master is bootstrapping. Could =0A= >> a throttling in the number of transitions be enforced at partition =0A= >> level? Could I somehow specify that a state with a slave =0A= >> and a bootstrapping node is undesirable? =0A= >> =0A= >> As a note=2C I have also looked at the RSync-replicateed filesystem =0A= >> example. The reason for not using the OfflineOnline or the =0A= >> MasterSlave model in my application is that I would like the =0A= >> bootstrapping node to receive updates from clients=2C i.e. be visible = =0A= >> during the bootstrap. For this reason=2C I am introducing the new =0A= >> BOOTSTRAPPING phase in-between OFFLINE and SLAVE. =0A= >> =0A= >> Regards=2C =0A= >> Vlad =0A= >> =0A= >> =0A= >> PS: The state model definition is as follows: =0A= >> =0A= >> builder.addState(MASTER=2C 1)=3B =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(SLAVE=2C 2)=3B=C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(BOOTSTRAP=2C 3)=3B=C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(OFFLINE)=3B =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(DROPPED)=3B =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 // Set the initial state when the node starts=C1= =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.initialState(OFFLINE)=3B =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 // Add transitions between the states. =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(OFFLINE=2C BOOTSTRAP=2C 4)= =3B=C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(BOOTSTRAP=2C SLAVE=2C 5)= =3B=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(SLAVE=2C MASTER=2C 6)=3B = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(MASTER=2C SLAVE=2C 3)=3B = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(SLAVE=2C OFFLINE=2C 2)=3B= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(OFFLINE=2C DROPPED=2C 1)= =3B=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 // set constraints on states.=C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 // static constraint =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.upperBound(MASTER=2C 1)=3B =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 // dynamic constraint=2C R means it should be de= rived based =0A= >> on the replication =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 // factor. =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =0A= >> =0A= >> =C1 =C1 =C1 =C1 =C1 =C1 builder.dynamicUpperBound(SLAVE=2C "R")=3B=C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =0A= >> =0A= >> =0A= >> =0A= >> =0A= > =0A= > =0A= > =0A= > =0A= =