Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A737A11B08 for ; Wed, 9 Apr 2014 19:42:13 +0000 (UTC) Received: (qmail 2249 invoked by uid 500); 9 Apr 2014 19:42:12 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 2219 invoked by uid 500); 9 Apr 2014 19:42:12 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 2207 invoked by uid 99); 9 Apr 2014 19:42:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 19:42:10 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of g.kishore@gmail.com designates 209.85.212.177 as permitted sender) Received: from [209.85.212.177] (HELO mail-wi0-f177.google.com) (209.85.212.177) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 19:42:03 +0000 Received: by mail-wi0-f177.google.com with SMTP id cc10so3836169wib.16 for ; Wed, 09 Apr 2014 12:41:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=q3mCRIGCc93PtVcsTgzd9A5sz+AHjDrBOGGPTTlAwcY=; b=mdzomdWBtMdlUbo0xYGm2WhVenwc9OQ9LbwqFRTcXB+9EI4P1I7kKbSRcnw6LVyFRM j/7H1oWYGCv/waswVzzE1c4LswqNBBkyHFb+qVLsQ72RBInTGYilyq5Sm2B4e4zb6LlO n/lUWRNBgOp2ORbE+6MLFoddb6jruBMwSNbHX5asD5WQgrnooG0g9Id52p5mBv2hdk8f p7Ug/Cf9DNneWbvvK7JGIHvnNaT/cbmXlTQ206v/E2QgUOYjdaahsjQ24pGLfYKyZxJV VuBdhe9o8lZ9QXqF6HIeh3DodMnx7vfqOVqMq1Wjw7MWmB2wizOYtYgh8GLoGBYgQ7D/ 7dAg== MIME-Version: 1.0 X-Received: by 10.180.93.133 with SMTP id cu5mr11462487wib.47.1397072501278; Wed, 09 Apr 2014 12:41:41 -0700 (PDT) Received: by 10.194.47.196 with HTTP; Wed, 9 Apr 2014 12:41:41 -0700 (PDT) In-Reply-To: References: <61F5E811-432F-4E18-B4AE-C49C04EC0123@gmail.com> Date: Wed, 9 Apr 2014 12:41:41 -0700 Message-ID: Subject: Re: keeping the master node up during bootstrap From: kishore g To: "user@helix.apache.org" Content-Type: multipart/alternative; boundary=f46d043890452a0ff304f6a14a23 X-Virus-Checked: Checked by ClamAV on apache.org --f46d043890452a0ff304f6a14a23 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable Hi Vlad, I have some questions. Can you join the IRC channel #apachehelix. thanks, Kishore G On Wed, Apr 9, 2014 at 11:35 AM, vlad.gm@gmail.com wrote= : > Upon some further testing, it seems that the controller does not execute > the events in the right sequence. > > Here are the results of some of my testing. Assume that we have a > partition NEWPROFILE_5 with the ideal state: > > "NEWPROFILE_5" : { > > "pf1.apps-pf.dev.docker_12000" : "SLAVE", > > "pf2.apps-pf.dev.docker_12000" : "MASTER" > > } > > I boot the host pf1 and a few minutes later the host pf2. In the > controller logs I see, when doing a grep for NEWPROFILE_5: > > 2014-04-08 17:04:35,309 (Thread-2) TaskAssignmentStage INFO: Sending > Message 69b4eddf-ac5f-4726-9d6b-bac742ad082e to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE to:MASTER > > 2014-04-08 17:27:08,187 (Thread-2) TaskAssignmentStage INFO: Sending > Message a221b1ac-0807-425e-9062-6507e45b0bfb to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:OFFLINE > to:BOOTSTRAP > > 2014-04-08 17:27:10,164 (Thread-2) TaskAssignmentStage INFO: Sending > Message 73ed85fd-49c9-46a5-b262-687d612c7d06 to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:BOOTSTRAP to:SL= AVE > > 2014-04-08 17:27:11,868 (Thread-2) TaskAssignmentStage INFO: Sending > Message fb21aecc-68cf-4b9f-9718-aa6ed535c29d to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE to:MASTER > > 2014-04-08 17:28:22,978 (Thread-2) TaskAssignmentStage INFO: Sending > Message ea441d18-b1f3-4ceb-96a2-3262cab1dfbe to > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:OFFLINE > to:BOOTSTRAP > > 2014-04-08 17:28:22,978 (Thread-2) TaskAssignmentStage INFO: Sending > Message f36b4d64-c790-413b-b9fa-915b9539d28c to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:MASTER to:SLAVE > > 2014-04-08 17:28:26,065 (Thread-2) TaskAssignmentStage INFO: Sending > Message 201429e1-e810-4017-b3ef-fb5930ac2192 to > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:BOOTSTRAP to:SL= AVE > > 2014-04-08 17:28:28,238 (Thread-2) TaskAssignmentStage INFO: Sending > Message 4a1fb64c-1063-4e49-a995-946d2dd25733 to > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE to:MASTER > > That is, the controller issues an offline->bootstrap command to pf-2, but > then issues a master->slave command to of-1 before bringing pf-2 up as a > slave as well (the last step before promotion to master). Since the > bootstrap->slave that follows takes time, the system spends time without = a > master for the partition. > > The state model definition was: > public static StateModelDefinition defineStateModel() { > StateModelDefinition.Builder builder =3D > new StateModelDefinition.Builder(KVHelixDefinitions.STATE_MODEL_NAME); > // Add states and their rank to indicate priority. Lower the rank higher > the > // priority > builder.addState(MASTER, 1); > builder.addState(SLAVE, 2); > builder.addState(BOOTSTRAP, 3); > builder.addState(OFFLINE); > builder.addState(DROPPED); > // Set the initial state when the node starts > builder.initialState(OFFLINE); > > // Add transitions between the states. > builder.addTransition(OFFLINE, BOOTSTRAP, 3); > builder.addTransition(BOOTSTRAP, SLAVE, 2); > builder.addTransition(SLAVE, MASTER, 1); > builder.addTransition(MASTER, SLAVE, 4); > builder.addTransition(SLAVE, OFFLINE, 5); > builder.addTransition(OFFLINE, DROPPED, 6); > > // set constraints on states. > // static constraint > builder.upperBound(MASTER, 1); > // dynamic constraint, R means it should be derived based on the > replication > // factor. > builder.dynamicUpperBound(SLAVE, "R"); > > StateModelDefinition statemodelDefinition =3D builder.build(); > > assert(statemodelDefinition.isValid()); > > return statemodelDefinition; > } > > I have tried reversing the values of the transition priorities. In this > case, the controller log file looked as follows: > > 2014-04-09 11:17:52,831 (Thread-2) TaskAssignmentStage INFO: Sending > Message 2b29a319-c1c6-4042-b1ad-3e3c1b5092a7 to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:OFFLINE > to:BOOTSTRAP > > 2014-04-09 11:17:55,672 (Thread-2) MessageGenerationStage INFO: Message > hasn't been removed for pf1.apps-pf.dev.docker_12000 to transitNEWPROFILE= _5 > to BOOTSTRAP, desiredState: MASTER > > 2014-04-09 11:17:57,047 (Thread-2) TaskAssignmentStage INFO: Sending > Message b1ca701d-65f1-46b9-9ae4-286400d6d266 to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:BOOTSTRAP to:SL= AVE > > 2014-04-09 11:17:58,888 (Thread-2) TaskAssignmentStage INFO: Sending > Message fe10228f-8f5b-4133-964a-5f6c7e60b0e6 to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE to:MASTER > > 2014-04-09 11:23:26,117 (Thread-2) TaskAssignmentStage INFO: Sending > Message 6252a4e6-0ab8-490a-a51d-c47195c434b5 to > pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:MASTER to:SLAVE > > 2014-04-09 11:23:26,117 (Thread-2) TaskAssignmentStage INFO: Sending > Message 18bbf028-cb51-4162-8226-a6564a121986 to > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:OFFLINE > to:BOOTSTRAP > > 2014-04-09 11:23:33,462 (Thread-2) MessageGenerationStage INFO: Message > hasn't been removed for pf2.apps-pf.dev.docker_12000 to transitNEWPROFILE= _5 > to BOOTSTRAP, desiredState: MASTER > > 2014-04-09 11:23:33,892 (Thread-2) TaskAssignmentStage INFO: Sending > Message c7fc4983-9d71-4dc4-bfee-2ad69e4de411 to > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:BOOTSTRAP to:SL= AVE > > 2014-04-09 11:23:35,933 (Thread-2) TaskAssignmentStage INFO: Sending > Message 75e715ed-3d53-4e39-b1e7-44695e4bfa03 to > pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5|[] from:SLAVE to:MASTER > > That is, the transition for master->slave for pf1 was executed before > taking any action on pf2, clearly the opposite of the right order. > > > On Tue, Apr 8, 2014 at 2:19 PM, Kanak Biscuitwala wr= ote: > >> >> Looks good, thanks for sharing! >> >> Kanak >> ________________________________ >> > Date: Tue, 8 Apr 2014 14:08:28 -0700 >> > Subject: Re: keeping the master node up during bootstrap >> > From: vlad.gm@gmail.com >> > To: user@helix.apache.org >> > >> > My modified code looks like: >> > >> > /* Setup a Helix cluster for the KVStore */ >> > public static void setupCluster() { >> > assert(cluster !=3D null); >> > clusterSetup.addCluster(cluster, true); >> > >> > =C1 =C1 =C1 =C1 ConstraintItemBuilder constraintItemBuilder =3D new >> > ConstraintItemBuilder(); >> > >> > =C1 =C1 =C1 =C1 constraintItemBuilder >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > .addConstraintAttribute(ConstraintAttribute.MESSAGE_TYPE.toString(), >> > "STATE_TRANSITION") >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > .addConstraintAttribute(ConstraintAttribute.PARTITION.toString(), ".*"= ) >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > .addConstraintAttribute(ConstraintAttribute.CONSTRAINT_VALUE.toString(= ), >> > "1"); >> > >> > =C1 =C1 =C1 =C1 clusterSetup.getClusterManagementTool().setConstraint(= cluster, >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 ClusterConstraints.ConstraintType.MESS= AGE_CONSTRAINT, >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 "constraint1", constraintItemBuilder.b= uild()); >> > =C1 =C1 } >> > >> > I will try to see whether it works in every situation. >> > >> > Regards, >> > Vlad >> > >> > >> > On Tue, Apr 8, 2014 at 8:59 AM, Vlad Balan >> > > wrote: >> > Hi Kishore, >> > >> > I managed to implement the bootstrapping using the constraint and it >> > appears to be running as expected. I will post my code shortly. >> > >> > Regards, >> > Vlad >> > >> > On Apr 8, 2014, at 8:27 AM, kishore g >> > > wrote: >> > >> > Hi Vlad, >> > >> > Did you get a chance to play with the constraint.=C1 I can write a sam= ple >> > code today to try this. >> > >> > Thanks, >> > Kishore G >> > >> > >> > On Thu, Apr 3, 2014 at 5:45 PM, >> > vlad.gm@gmail.com >> > > wrote: >> > >> > Thank you Kanak and Kishore! I will try enforcing the per-partition >> > constraint and let you know if somehow it does not work. I was looking >> > at the throttling documentation, but somehow missed that a >> > per-partition constraint was an option! >> > >> > Regards, >> > Vlad >> > >> > >> > On Thu, Apr 3, 2014 at 5:42 PM, kishore g >> > > wrote: >> > Hi Vlad, >> > >> > You can try setting the transition priority order and a constraint tha= t >> > there should be only one transition per partition across the cluster. >> > >> > So the transition priority could be something like >> > >> > Slave-Master >> > Offfline -> Bootstrap >> > Bootstrap->Slave >> > Slave->Master >> > >> > For the rest not sure if order matters. >> > >> > Also set the max transitions constraint to 1 per partition. >> > >> > The reason I put Slave-Master before Offline->Bootstrap is to ensure >> > that availability is given more importance. For example if you have 3 >> > nodes, N1, N2, N3. N1 is Master, N2 is Slave, and N3 is down. If N1 >> > goes down and N3 comes up at the same time. We probably dont want to >> > wait for N3 to bootstrap before promoting N2 to Master. >> > >> > I haven't tested this but assuming the constraints enforcement works, >> > this should do the trick. >> > >> > Does this make sense? Let me know if this does not work, we can add a >> > test case. >> > >> > thanks, >> > Kishore G >> > >> > >> > >> > >> > >> > >> > On Thu, Apr 3, 2014 at 4:57 PM, >> > vlad.gm@gmail.com >> > > wrote: >> > >> > Dear all, >> > >> > I am trying to construct a state model with the following transition >> diagram: >> > >> > OFFLINE -> BOOTSTRAPPING <---> SLAVE <-----> MASTER >> > =C1 =C1 =C1 =C1 =C1<----------------------------------- >> > >> > That is, an offline mode can go into a bootstraping state, from the >> > bootstrap state it can go into a slave state, >> > from slave it can go from master, from master to slave and from slave >> > it can go offline. >> > >> > Assume that if I have a partition with two nodes pf1 and pf2 and a >> > partition partition_0 with the following ideal state: >> > >> > partition_0: pf2: MASTER pf1: SLAVE, >> > >> > and that currently pf1 is serving as a master. When pf2 boots, Helix >> > will issue, almost simultaneously, two commands: >> > for pf1: transition from MASTER to SLAVE >> > for pf2: transition from BOOTSTRAPPING to SLAVE >> > >> > My understanding is that this happens since Helix is trying to execute >> > as many commands in parallel and since the last state >> > has pf2 as master. However, the transition from BOOTSTRAPPING to SLAVE >> > for pf2 involves a long data copy step, so >> > I would like to keep pf1 as a master in the meanwhile. I tried >> > prioritizing the transition from BOOTSTRAPPING to SLAVE >> > over the transition from MASTER to SLAVE, however Helix still issues >> > them in parallel (as it should). >> > >> > I was wondering what my options would be in order to keep the master u= p >> > while the future master is bootstrapping. Could >> > a throttling in the number of transitions be enforced at partition >> > level? Could I somehow specify that a state with a slave >> > and a bootstrapping node is undesirable? >> > >> > As a note, I have also looked at the RSync-replicateed filesystem >> > example. The reason for not using the OfflineOnline or the >> > MasterSlave model in my application is that I would like the >> > bootstrapping node to receive updates from clients, i.e. be visible >> > during the bootstrap. For this reason, I am introducing the new >> > BOOTSTRAPPING phase in-between OFFLINE and SLAVE. >> > >> > Regards, >> > Vlad >> > >> > >> > PS: The state model definition is as follows: >> > >> > builder.addState(MASTER, 1); =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(SLAVE, 2);=C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(BOOTSTRAP, 3);=C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(OFFLINE); =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(DROPPED); =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 // Set the initial state when the node starts= =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.initialState(OFFLINE); =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 // Add transitions between the states. =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(OFFLINE, BOOTSTRAP, 4);= =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(BOOTSTRAP, SLAVE, 5);=C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(SLAVE, MASTER, 6); =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(MASTER, SLAVE, 3); =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(SLAVE, OFFLINE, 2);=C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(OFFLINE, DROPPED, 1);=C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 // set constraints on states.=C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 // static constraint =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.upperBound(MASTER, 1); =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 // dynamic constraint, R means it should be de= rived based >> > on the replication =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 // factor. =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 >> > >> > =C1 =C1 =C1 =C1 =C1 =C1 builder.dynamicUpperBound(SLAVE, "R");=C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 >> > >> > >> > >> > >> >> > > --f46d043890452a0ff304f6a14a23 Content-Type: text/html; charset=KOI8-R Content-Transfer-Encoding: quoted-printable
Hi Vlad,

I have some questions. Can you = join the IRC channel #apachehelix.

thanks,
Kishore G


On W= ed, Apr 9, 2014 at 11:35 AM, vlad.gm@g= mail.com <vlad.gm@gmail.com> wrote:
Upon some further testing, = it seems that the controller does not execute the events in the right seque= nce.

Here are the results of some of my testing. Assume that we h= ave a partition NEWPROFILE_5 with the ideal state:

=9A"NE= WPROFILE_5" : {

=9A =9A =9A &quo= t;pf1.apps-pf.dev.docker_12000" : "SLAVE",

=9A =9A =9A &quo= t;pf2.apps-pf.dev.docker_12000" : "MASTER"

=9A =9A }


I boot the host pf1 and a few minutes later the host= pf2. In the controller logs I see, when doing a grep for NEWPROFILE_5:

2014-04-08 = 17:04:35,309 (Thread-2) TaskAssignmentStage INFO: Sending Message 69b4eddf-= ac5f-4726-9d6b-bac742ad082e to pf1.apps-pf.dev.docker_12000 transit NEWPROF= ILE_5|[] from:SLAVE to:MASTER

2014-04-08 17:27= :08,187 (Thread-2) TaskAssignmentStage INFO: Sending Message a221b1ac-0807-= 425e-9062-6507e45b0bfb to pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:OFFLINE to:BOOTSTRAP

2014-04-08 17:27= :10,164 (Thread-2) TaskAssignmentStage INFO: Sending Message 73ed85fd-49c9-= 46a5-b262-687d612c7d06 to pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:BOOTSTRAP to:SLAVE

2014-04-08 17:27= :11,868 (Thread-2) TaskAssignmentStage INFO: Sending Message fb21aecc-68cf-= 4b9f-9718-aa6ed535c29d to pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:SLAVE to:MASTER

2014-04-08 17:28= :22,978 (Thread-2) TaskAssignmentStage INFO: Sending Message ea441d18-b1f3-= 4ceb-96a2-3262cab1dfbe to pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:OFFLINE to:BOOTSTRAP

2014-04-08 17:28= :22,978 (Thread-2) TaskAssignmentStage INFO: Sending Message f36b4d64-c790-= 413b-b9fa-915b9539d28c to pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:MASTER to:SLAVE

2014-04-08 17:28= :26,065 (Thread-2) TaskAssignmentStage INFO: Sending Message 201429e1-e810-= 4017-b3ef-fb5930ac2192 to pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:BOOTSTRAP to:SLAVE

2014-04-08 17:28= :28,238 (Thread-2) TaskAssignmentStage INFO: Sending Message 4a1fb64c-1063-= 4e49-a995-946d2dd25733 to pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:SLAVE to:MASTER


That is, the controller issues an offline->boo= tstrap command to pf-2, but then issues a master->slave command to of-1 = before bringing pf-2 up as a slave as well (the last step before promotion = to master). Since the bootstrap->slave that follows takes time, the syst= em spends time without a master for the partition.

The state model definition was:
public s= tatic StateModelDefinition defineStateModel() {
StateModelDefinition.Builder builder =3D
new StateModelDefiniti= on.Builder(KVHelixDefinitions.STATE_MODEL_NAME);
// Add states and their rank to indicate pri= ority. Lower the rank higher the
// priority
builder.addSt= ate(MASTER, 1);
bu= ilder.addState(SLAVE, 2);
builder.addState(BOOTST= RAP, 3);
builder.a= ddState(OFFLINE);
= builder.addState(DROPPED);
/= / Set the initial state when the node starts
builder.initialState(OFFLINE);

// Add t= ransitions between the states.
builder.addTransi= tion(OFFLINE, BOOTSTRAP, 3);
builder.addTransition(BOOTSTRAP, SLAVE, 2);
builder.addTransition(SLAVE, MASTER, 1= );
builder.addTransition(M= ASTER, SLAVE, 4);
= builder.addTransition(SLAVE, OFFLINE, 5);
builder.addTransition(OFFLINE, DROPPED, 6);

// set c= onstraints on states.
// static constraint
= builder.upperBound(MASTER, 1);
/= / dynamic constraint, R means it should be derived based on the replication=
// factor.
builder.dynamicUpperBou= nd(SLAVE, "R");

St= ateModelDefinition statemodelDefinition =3D builder.build();

=
assert(statemodel= Definition.isValid());

return s= tatemodelDefinition;
}

I have tried reversing the values of the = transition priorities. In this case, the controller log file looked as foll= ows:

2014-04-09 = 11:17:52,831 (Thread-2) TaskAssignmentStage INFO: Sending Message 2b29a319-= c1c6-4042-b1ad-3e3c1b5092a7 to pf1.apps-pf.dev.docker_12000 transit NEWPROF= ILE_5|[] from:OFFLINE to:BOOTSTRAP

2014-04-09 11:17= :55,672 (Thread-2) MessageGenerationStage INFO: Message hasn't been rem= oved for pf1.apps-pf.dev.docker_12000 to transitNEWPROFILE_5 to BOOTSTRAP, = desiredState: MASTER

2014-04-09 11:17= :57,047 (Thread-2) TaskAssignmentStage INFO: Sending Message b1ca701d-65f1-= 46b9-9ae4-286400d6d266 to pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:BOOTSTRAP to:SLAVE

2014-04-09 11:17= :58,888 (Thread-2) TaskAssignmentStage INFO: Sending Message fe10228f-8f5b-= 4133-964a-5f6c7e60b0e6 to pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:SLAVE to:MASTER

2014-04-09 11:23= :26,117 (Thread-2) TaskAssignmentStage INFO: Sending Message 6252a4e6-0ab8-= 490a-a51d-c47195c434b5 to pf1.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:MASTER to:SLAVE

2014-04-09 11:23= :26,117 (Thread-2) TaskAssignmentStage INFO: Sending Message 18bbf028-cb51-= 4162-8226-a6564a121986 to pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:OFFLINE to:BOOTSTRAP

2014-04-09 11:23= :33,462 (Thread-2) MessageGenerationStage INFO: Message hasn't been rem= oved for pf2.apps-pf.dev.docker_12000 to transitNEWPROFILE_5 to BOOTSTRAP, = desiredState: MASTER

2014-04-09 11:23= :33,892 (Thread-2) TaskAssignmentStage INFO: Sending Message c7fc4983-9d71-= 4dc4-bfee-2ad69e4de411 to pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:BOOTSTRAP to:SLAVE

2014-04-09 11:23= :35,933 (Thread-2) TaskAssignmentStage INFO: Sending Message 75e715ed-3d53-= 4e39-b1e7-44695e4bfa03 to pf2.apps-pf.dev.docker_12000 transit NEWPROFILE_5= |[] from:SLAVE to:MASTER


That is, the transition for master->slave for = pf1 was executed before taking any action on pf2, clearly the opposite of t= he right order.


On Tue, Apr 8, 2014 at 2:19 PM, Kanak Biscuitwala <kanak.b@hotmail.com> wrote:

Looks good, thanks for sharing!

Kanak
________________________________
> Date: Tue, 8 Apr 2014 14:08:28 -0700
> Subject: Re: keeping the master node up during bootstrap
> From: vlad.gm@g= mail.com
> To: user@he= lix.apache.org
>
> My modified code looks like:
>
> /* Setup a Helix cluster for the KVStore */
> public static void setupCluster() {
> assert(cluster !=3D null);
> clusterSetup.addCluster(cluster, true);
>
> =C1 =C1 =C1 =C1 ConstraintItemBuilder constraintItemBuilder =3D = new
> ConstraintItemBuilder();
>
> =C1 =C1 =C1 =C1 constraintItemBuilder
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> .addConstraintAttribute(ConstraintAttribute.MESSAGE_TYPE.toStrin= g(),
> "STATE_TRANSITION")
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> .addConstraintAttribute(ConstraintAttribute.PARTITION.toString()= , ".*")
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> .addConstraintAttribute(ConstraintAttribute.CONSTRAINT_VALUE.toS= tring(),
> "1");
>
> =C1 =C1 =C1 =C1 clusterSetup.getClusterManagementTool().setConstraint(= cluster,
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 ClusterConstraints.ConstraintType.MESS= AGE_CONSTRAINT,
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 "constraint1", constraintIte= mBuilder.build());
> =C1 =C1 }
>
> I will try to see whether it works in every situation.
>
> Regards,
> Vlad
>
>
> On Tue, Apr 8, 2014 at 8:59 AM, Vlad Balan
> <= vlad.gm@gmail.com<mailto:vlad.gm@gmail.com>> wrote:
> Hi Kishore,
>
> I managed to implement the bootstrapping using the constraint and it > appears to be running as expected. I will post my code shortly.
>
> Regards,
> Vlad
>
> On Apr 8, 2014, at 8:27 AM, kishore g
> <g.kishore@gmail.com<mailto:g.kishore@gmail.com>> wrote:
>
> Hi Vlad,
>
> Did you get a chance to play with the constraint.=C1 I can write= a sample
> code today to try this.
>
> Thanks,
> Kishore G
>
>
> On Thu, Apr 3, 2014 at 5:45 PM,
> vlad.gm@g= mail.com<mailto:vlad.gm@gmail.com>
> <vlad.g= m@gmail.com<mailto:vlad.gm@gmail.com>> wrote:
>
> Thank you Kanak and Kishore! I will try enforcing the per-partition > constraint and let you know if somehow it does not work. I was looking=
> at the throttling documentation, but somehow missed that a
> per-partition constraint was an option!
>
> Regards,
> Vlad
>
>
> On Thu, Apr 3, 2014 at 5:42 PM, kishore g
> <g.kishore@gmail.com<mailto:g.kishore@gmail.com>> wrote:
> Hi Vlad,
>
> You can try setting the transition priority order and a constraint tha= t
> there should be only one transition per partition across the cluster.<= br> >
> So the transition priority could be something like
>
> Slave-Master
> Offfline -> Bootstrap
> Bootstrap->Slave
> Slave->Master
>
> For the rest not sure if order matters.
>
> Also set the max transitions constraint to 1 per partition.
>
> The reason I put Slave-Master before Offline->Bootstrap is to ensur= e
> that availability is given more importance. For example if you have 3<= br> > nodes, N1, N2, N3. N1 is Master, N2 is Slave, and N3 is down. If N1 > goes down and N3 comes up at the same time. We probably dont want to > wait for N3 to bootstrap before promoting N2 to Master.
>
> I haven't tested this but assuming the constraints enforcement wor= ks,
> this should do the trick.
>
> Does this make sense? Let me know if this does not work, we can add a<= br> > test case.
>
> thanks,
> Kishore G
>
>
>
>
>
>
> On Thu, Apr 3, 2014 at 4:57 PM,
> vlad.gm@g= mail.com<mailto:vlad.gm@gmail.com>
> <vlad.g= m@gmail.com<mailto:vlad.gm@gmail.com>> wrote:
>
> Dear all,
>
> I am trying to construct a state model with the following transition d= iagram:
>
> OFFLINE -> BOOTSTRAPPING <---> SLAVE <-----> MASTER
> =C1 =C1 =C1 =C1 =C1<-----------------------------------
>
> That is, an offline mode can go into a bootstraping state, from the > bootstrap state it can go into a slave state,
> from slave it can go from master, from master to slave and from slave<= br> > it can go offline.
>
> Assume that if I have a partition with two nodes pf1 and pf2 and a
> partition partition_0 with the following ideal state:
>
> partition_0: pf2: MASTER pf1: SLAVE,
>
> and that currently pf1 is serving as a master. When pf2 boots, Helix > will issue, almost simultaneously, two commands:
> for pf1: transition from MASTER to SLAVE
> for pf2: transition from BOOTSTRAPPING to SLAVE
>
> My understanding is that this happens since Helix is trying to execute=
> as many commands in parallel and since the last state
> has pf2 as master. However, the transition from BOOTSTRAPPING to SLAVE=
> for pf2 involves a long data copy step, so
> I would like to keep pf1 as a master in the meanwhile. I tried
> prioritizing the transition from BOOTSTRAPPING to SLAVE
> over the transition from MASTER to SLAVE, however Helix still issues > them in parallel (as it should).
>
> I was wondering what my options would be in order to keep the master u= p
> while the future master is bootstrapping. Could
> a throttling in the number of transitions be enforced at partition
> level? Could I somehow specify that a state with a slave
> and a bootstrapping node is undesirable?
>
> As a note, I have also looked at the RSync-replicateed filesystem
> example. The reason for not using the OfflineOnline or the
> MasterSlave model in my application is that I would like the
> bootstrapping node to receive updates from clients, i.e. be visible > during the bootstrap. For this reason, I am introducing the new
> BOOTSTRAPPING phase in-between OFFLINE and SLAVE.
>
> Regards,
> Vlad
>
>
> PS: The state model definition is as follows:
>
> builder.addState(MASTER, 1); =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(SLAVE, 2);=C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(BOOTSTRAP, 3);=C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(OFFLINE); =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addState(DROPPED); =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 // Set the initial state when the node starts= =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.initialState(OFFLINE); =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 // Add transitions between the states. =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(OFFLINE, BOOTSTRAP, 4);= =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(BOOTSTRAP, SLAVE, 5);=C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(SLAVE, MASTER, 6); =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(MASTER, SLAVE, 3); =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(SLAVE, OFFLINE, 2);=C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.addTransition(OFFLINE, DROPPED, 1);=C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 > =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 // set constraints on states.=C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 // static constraint =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.upperBound(MASTER, 1); =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 // dynamic constraint, R means it = should be derived based
> on the replication =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 // factor. =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1= =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1=
> =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 = =C1
>
> =C1 =C1 =C1 =C1 =C1 =C1 builder.dynamicUpperBound(SLAVE, "R"= );=C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1 =C1
>
>
>
>
=9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A= =9A =9A


--f46d043890452a0ff304f6a14a23--