Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5616D11B0B for ; Fri, 4 Apr 2014 00:45:33 +0000 (UTC) Received: (qmail 7523 invoked by uid 500); 4 Apr 2014 00:45:32 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 7434 invoked by uid 500); 4 Apr 2014 00:45:32 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 7427 invoked by uid 99); 4 Apr 2014 00:45:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2014 00:45:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vlad.gm@gmail.com designates 209.85.160.178 as permitted sender) Received: from [209.85.160.178] (HELO mail-yk0-f178.google.com) (209.85.160.178) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2014 00:45:25 +0000 Received: by mail-yk0-f178.google.com with SMTP id 79so2338421ykr.9 for ; Thu, 03 Apr 2014 17:45:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=8lOKEQZplPRLPdzhOEl39xg6vpvMF+d1RWddxroDa9Y=; b=REsWQCDav+IZWiTzMN9VTlswHCmDPi4lcKoG7JvJfFN33O8OXh7UVT2TQSLfDcNY5E SS7npO42Xg8k3m96xH5ZOxVXNynnhe28pCZU1Itruv1+MgEe8d29jHBTsIQui1XeGyD2 i3U24H3mCQsyA/UWzeBAMhfESpAUpVu8AdR2SwKuCFnXPXUhf0PYbjc3w1u6VnO664hV 5r7qfFjGz/SeWWAQRypGq9HS+Az0KY/S5trA0Us0ums8y7wkzvRE9JZme1dsiuL39XjX VoCWoC7IddeDbNNnzFpJrb+t7LTlxLozsclIKZo9Q3pVJPdBmLrdoNXH3A/d3KerTl+g UGqA== MIME-Version: 1.0 X-Received: by 10.236.128.180 with SMTP id f40mr6105307yhi.71.1396572303398; Thu, 03 Apr 2014 17:45:03 -0700 (PDT) Received: by 10.170.148.8 with HTTP; Thu, 3 Apr 2014 17:45:03 -0700 (PDT) In-Reply-To: References: Date: Thu, 3 Apr 2014 17:45:03 -0700 Message-ID: Subject: Re: keeping the master node up during bootstrap From: "vlad.gm@gmail.com" To: user@helix.apache.org Content-Type: multipart/alternative; boundary=20cf300fb4c70c1af204f62cd40d X-Virus-Checked: Checked by ClamAV on apache.org --20cf300fb4c70c1af204f62cd40d Content-Type: text/plain; charset=ISO-8859-1 Thank you Kanak and Kishore! I will try enforcing the per-partition constraint and let you know if somehow it does not work. I was looking at the throttling documentation, but somehow missed that a per-partition constraint was an option! Regards, Vlad On Thu, Apr 3, 2014 at 5:42 PM, kishore g wrote: > Hi Vlad, > > You can try setting the transition priority order and a constraint that > there should be only one transition per partition across the cluster. > > So the transition priority could be something like > > Slave-Master > Offfline -> Bootstrap > Bootstrap->Slave > Slave->Master > > For the rest not sure if order matters. > > Also set the max transitions constraint to 1 per partition. > > The reason I put Slave-Master before Offline->Bootstrap is to ensure that > availability is given more importance. For example if you have 3 nodes, N1, > N2, N3. N1 is Master, N2 is Slave, and N3 is down. If N1 goes down and N3 > comes up at the same time. We probably dont want to wait for N3 to > bootstrap before promoting N2 to Master. > > I haven't tested this but assuming the constraints enforcement works, this > should do the trick. > > Does this make sense? Let me know if this does not work, we can add a test > case. > > thanks, > Kishore G > > > > > > > On Thu, Apr 3, 2014 at 4:57 PM, vlad.gm@gmail.com wrote: > >> >> Dear all, >> >> I am trying to construct a state model with the following transition >> diagram: >> >> OFFLINE -> BOOTSTRAPPING <---> SLAVE <-----> MASTER >> <----------------------------------- >> >> That is, an offline mode can go into a bootstraping state, from the >> bootstrap state it can go into a slave state, >> from slave it can go from master, from master to slave and from slave it >> can go offline. >> >> Assume that if I have a partition with two nodes pf1 and pf2 and a >> partition partition_0 with the following ideal state: >> >> partition_0: pf2: MASTER pf1: SLAVE, >> >> and that currently pf1 is serving as a master. When pf2 boots, Helix will >> issue, almost simultaneously, two commands: >> for pf1: transition from MASTER to SLAVE >> for pf2: transition from BOOTSTRAPPING to SLAVE >> >> My understanding is that this happens since Helix is trying to execute as >> many commands in parallel and since the last state >> has pf2 as master. However, the transition from BOOTSTRAPPING to SLAVE >> for pf2 involves a long data copy step, so >> I would like to keep pf1 as a master in the meanwhile. I tried >> prioritizing the transition from BOOTSTRAPPING to SLAVE >> over the transition from MASTER to SLAVE, however Helix still issues them >> in parallel (as it should). >> >> I was wondering what my options would be in order to keep the master up >> while the future master is bootstrapping. Could >> a throttling in the number of transitions be enforced at partition level? >> Could I somehow specify that a state with a slave >> and a bootstrapping node is undesirable? >> >> As a note, I have also looked at the RSync-replicateed filesystem >> example. The reason for not using the OfflineOnline or the >> MasterSlave model in my application is that I would like the >> bootstrapping node to receive updates from clients, i.e. be visible >> during the bootstrap. For this reason, I am introducing the new >> BOOTSTRAPPING phase in-between OFFLINE and SLAVE. >> >> Regards, >> Vlad >> >> >> PS: The state model definition is as follows: >> >> builder.addState(MASTER, 1); >> >> >> >> builder.addState(SLAVE, 2); >> >> >> >> builder.addState(BOOTSTRAP, 3); >> >> >> >> builder.addState(OFFLINE); >> >> >> >> builder.addState(DROPPED); >> >> >> >> // Set the initial state when the node starts >> >> >> >> builder.initialState(OFFLINE); >> >> >> >> >> >> >> >> // Add transitions between the states. >> >> >> >> builder.addTransition(OFFLINE, BOOTSTRAP, 4); >> >> >> >> builder.addTransition(BOOTSTRAP, SLAVE, 5); >> >> >> >> builder.addTransition(SLAVE, MASTER, 6); >> >> >> >> builder.addTransition(MASTER, SLAVE, 3); >> >> >> >> builder.addTransition(SLAVE, OFFLINE, 2); >> >> >> >> builder.addTransition(OFFLINE, DROPPED, 1); >> >> >> >> >> >> >> >> // set constraints on states. >> >> >> >> // static constraint >> >> >> >> builder.upperBound(MASTER, 1); >> >> >> >> // dynamic constraint, R means it should be derived based on >> the replication >> >> >> // factor. >> >> >> >> builder.dynamicUpperBound(SLAVE, "R"); >> > > --20cf300fb4c70c1af204f62cd40d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Thank you Kanak and Kishore! I will try enforcing= the per-partition constraint and let you know if somehow it does not work.= I was looking at the throttling documentation, but somehow missed that a p= er-partition constraint was an option!

Regards,
Vlad


On Thu, Apr 3, 2014 at 5:42 PM, k= ishore g <g.kishore@gmail.com> wrote:
Hi Vlad,

You can try setting the transition priority o= rder and a constraint that there should be only one transition per partitio= n across the cluster.

So the transition priority could be something like

Slave-Maste= r
Offfline -> Bootstrap
Bootstrap->Slave
Slave->Master
For the rest not sure if order matters.

Also set the max= transitions constraint to 1 per partition.

The reason I put Slave-Master before Offline->Bootstrap i= s to ensure that availability is given more importance. For example if you = have 3 nodes, N1, N2, N3. N1 is Master, N2 is Slave, and N3 is down. If N1 = goes down and N3 comes up at the same time. We probably dont want to wait f= or N3 to bootstrap before promoting N2 to Master.

I haven't tested this but assuming the constraints= enforcement works, this should do the trick.

Does this make s= ense? Let me know if this does not work, we can add a test case.

thanks,
Kishore G






On Thu, Apr 3, 2= 014 at 4:57 PM, vlad= .gm@gmail.com <vlad.gm@gmail.com> wrote:

Dear all,

I am trying to construct a state model with the following = transition diagram:

OFFLINE -> BOOTSTRAPPING <---> SLAVE <-----= > MASTER
=A0 =A0 =A0 =A0 =A0<-----------------------------------
<= br>
That is, an offline mode can go into a bootstraping state, fr= om the bootstrap state it can go into a slave state,
from slave i= t can go from master, from master to slave and from slave it can go offline= .

Assume that if I have a partition with two nodes pf1 an= d pf2 and a partition partition_0 with the following ideal state:

partition_0: pf2: MASTER pf1: SLAVE,

and that currently pf1 is serving as a master. When pf2 boots, Helix w= ill issue, almost simultaneously, two commands:
for pf1: transiti= on from MASTER to SLAVE
for pf2: transition from BOOTSTRAPPING to= SLAVE

My understanding is that this happens since Helix is tr= ying to execute as many commands in parallel and since the last state
=
has pf2 as master. However, the transition from BOOTSTRAPPING to SLAVE= for pf2 involves a long data copy step, so
I would like to keep pf1 as a master in the meanwhile. I tried priorit= izing the transition from BOOTSTRAPPING to SLAVE
over the transit= ion from MASTER to SLAVE, however Helix still issues them in parallel (as i= t should).

I was wondering what my options would be in order to ke= ep the master up while the future master is bootstrapping. Could
= a throttling in the number of transitions be enforced at partition level? C= ould I somehow specify that a state with a slave
and a bootstrapping node is undesirable?

As a= note, I have also looked at the RSync-replicateed filesystem example. The = reason for not using the OfflineOnline or the
MasterSlave model i= n my application is that I would like the bootstrapping node to receive upd= ates from clients, i.e. be visible
during the bootstrap. For this reason, I am introducing the new BOOTST= RAPPING phase in-between OFFLINE and SLAVE.

Regard= s,
Vlad


PS: The state mod= el definition is as follows:

builder.addState(MASTER, 1); =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addState(SLAVE, 2);=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=

=A0 =A0 =A0 =A0 =A0 =A0 builder.addState(BOOTSTRAP, 3= );=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addState(OFFLINE); =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addState(DROPPED); =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 // Set the initial state when= the node starts=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.initialState(OFFLINE); =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 // Add transitions between th= e states. =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addTransition(OFFLINE, BOOTSTRAP, = 4);=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addTransition(BOOTSTRAP, SLAVE, 5);=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addTransition(SLAVE, MASTER, 6); =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addTransition(MASTER, SLAVE, 3); =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addTransition(SLAVE, OFFLINE, 2);=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.addTransition(OFFLINE, DROPPED, 1);=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 // set constraints on states.= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 // static constraint = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.upperBound(MASTER, 1<= /span>); =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 // dynamic constraint, R mean= s it should be derived based on the replication =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0

=A0 =A0 =A0 =A0 =A0 =A0 // factor. =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0

=A0 =A0 =A0 =A0 =A0 =A0 builder.dynamicUpperBound(SLAVE, "R");=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=



--20cf300fb4c70c1af204f62cd40d--