Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: <CAM+wfv5uRfSMXhH04xXWR59VK499yhv_zHEf0oaRqhtcpP63mA@mail.gmail.com>
References: <CAM+wfv4Lbaouv=c4BXW41A7Rc71nvgw+DcE+mOQXtC3Y7bEn3A@mail.gmail.com>
 <1A9BB7DA-9D76-40CF-92C8-743A3B418743@apache.org> <CAM+wfv7h5MFeqsWpFevomi4ccjdwtQpQRfTfa8zcZye7L299UA@mail.gmail.com>
 <CAM+wfv7vihfDcoRqDmAuyEvxHxLVGBhKgEA+2YX5=-+2OAfJRg@mail.gmail.com> <CAM+wfv5uRfSMXhH04xXWR59VK499yhv_zHEf0oaRqhtcpP63mA@mail.gmail.com>
From: Michael Han <hanm@cloudera.com>
Date: Tue, 11 Oct 2016 15:46:23 -0700
Message-ID: <CA+i0x1+0bKgjGpz4BTm_0G=MXQ=XX9ctByNr1-QAvdVeWp6UdQ@mail.gmail.com>
Subject: Re: Zookeeper leader election takes a long time.
To: UserZooKeeper <user@zookeeper.apache.org>
Content-Type: multipart/alternative; boundary=001a11426a3c30bad6053e9ea74b
archived-at: Tue, 11 Oct 2016 22:47:06 -0000

--001a11426a3c30bad6053e9ea74b
Content-Type: text/plain; charset=UTF-8

Hi Anand,

>> We have isolated it to a test setup, where we are able
to reproduce this somewhat consistently if we keep a node powered off.

Do you mind share your setup / steps to reproduce if the setup only
involves ZooKeeper without other dependencies?


On Tue, Oct 11, 2016 at 2:56 PM, Anand Parthasarathy <
anpartha@avinetworks.com> wrote:

> Folks,
>
> Sending a quick note again to find out if there is any insight the
> community can offer in terms of a solution or workaround? We use zookeeper
> for service discovery in our product and this issue has surfaced in a large
> customer site a couple of times and we need to figure out a solution soon.
>
> Thanks,
> Anand.
>
> On Mon, Oct 10, 2016 at 10:15 AM, Anand Parthasarathy <
> anpartha@avinetworks.com> wrote:
>
> > Folks,
> >
> > Any insight into this or any workarounds that you can think of to
> mitigate
> > against this issue? We have isolated it to a test setup, where we are
> able
> > to reproduce this somewhat consistently if we keep a node powered off.
> >
> > Thanks,
> > Anand.
> >
> > On Sat, Oct 8, 2016 at 10:05 AM, Anand Parthasarathy <
> > anpartha@avinetworks.com> wrote:
> >
> >> Hi Flavio,
> >>
> >> I have attached the logs from node 1 and node 3. Node 2 was powered off
> >> around 10-03 12:36. Leader election kept going until 10-03 15:57:16
> when it
> >> finally converged.
> >>
> >> Thanks,
> >> Anand.
> >>
> >> On Sat, Oct 8, 2016 at 7:55 AM, Flavio Junqueira <fpj@apache.org>
> wrote:
> >>
> >>> Hi Anand,
> >>>
> >>> I don't understand whether 1 and 3 were able or even trying to connect
> >>> to each other. They should be able to elect a leader between them and
> make
> >>> progress. You might want to upload logs and let us know.
> >>>
> >>> -Flavio
> >>>
> >>> > On 08 Oct 2016, at 02:11, Anand Parthasarathy <
> >>> anpartha@avinetworks.com> wrote:
> >>> >
> >>> > Hi,
> >>> >
> >>> > We are currently using zookeeper 3.4.6 version and use a 3 node
> >>> solution in
> >>> > our system. We see that occasionally, when a node is powered off (in
> >>> this
> >>> > instance, it was actually a leader node), the remaining two nodes do
> >>> not
> >>> > form a quorum for a really long time. Looking at the logs, it appears
> >>> the
> >>> > sequence is as follows:
> >>> > - Node 2 is the zookeeper leader
> >>> > - Node 2 is powered off
> >>> > - Node 1 and Node 3 recognize and start the election
> >>> > - Node 3 times out after initLimit * tickTime with "Timeout while
> >>> waiting
> >>> > for quorum" for Round N
> >>> > - Node 1 times out after initLimit * tickTime with "Exception while
> >>> trying
> >>> > to follow leader" for Round N+1 at the same time.
> >>> > - And the process continues where N is sequentially incrementing.
> >>> > - This happens for a long time.
> >>> > - In one instance, we used tickTime=5000 and initLimit=20 and it took
> >>> > around 3.5 hours to converge.
> >>> > - In a given round, Node 1 will try connecting to Node 2, gets
> >>> connection
> >>> > refused waits for notification timeout which increases by 2 every
> >>> iteration
> >>> > until it hits the initLimit. Connection Refused is because the node 2
> >>> comes
> >>> > up after reboot, but zookeeper process is not started (due to a
> >>> different
> >>> > failure).
> >>> >
> >>> > It looks similar to ZOOKEEPER-2164 but there it is a connection
> timeout
> >>> > where Node 2 is not reachable.
> >>> >
> >>> > Could you pls. share if you have seen this issue and if so, what is
> the
> >>> > workaround that can be employed in 3.4.6.
> >>> >
> >>> > Thanks,
> >>> > Anand.
> >>>
> >>>
> >>
> >
>


-- 
Cheers
Michael.

--001a11426a3c30bad6053e9ea74b--