Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (athena.apache.org: domain of ted.dunning@gmail.com
 designates 209.85.216.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <9694A6C3D68A4249BD9E1A875B6BA81E05B1006D85@bos0ex01.corp.attivio.com>
References: 
 <9694A6C3D68A4249BD9E1A875B6BA81E05B0F090D7@bos0ex01.corp.attivio.com>
 <CAJwFCa3ggpQRMSDyo86w4569AK+8Nn-8_JXxrwLMs6-zXJBv4g@mail.gmail.com>
 <9694A6C3D68A4249BD9E1A875B6BA81E05B0F090E7@bos0ex01.corp.attivio.com>
 <9694A6C3D68A4249BD9E1A875B6BA81E05B1006D85@bos0ex01.corp.attivio.com>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Wed, 14 Sep 2011 20:46:05 +0000
Message-ID: 
 <CAJwFCa2z3k_=GZqJK9c54tKkOdZPz2F9sfBFEAM2MbpkqK_D2g@mail.gmail.com>
Subject: Re: disconnects and auto renewal
To: user@zookeeper.apache.org
Content-Type: multipart/alternative; boundary=bcaec51a8276861de604acecdab2

--bcaec51a8276861de604acecdab2
Content-Type: text/plain; charset=UTF-8

On Wed, Sep 14, 2011 at 8:16 PM, Martin Serrano <martin@attivio.com> wrote:

> ....  The original reason the code was there was to handle the case where a
> client is mainly used for listening to remote events.  So once it starts, it
> sets up a few watches and really doesn't interact with the server after
> that.  The thought was that if such a client was disconnected and did not
> handle that case, we'd never know about it and it would seem like no remote
> events occurred.


The issue here is that disconnects can occur for many reasons and are
generally rather benign.

For instance, if you are doing a rolling upgrade of the Zookeeper servers,
you will get a disconnect and a quick reconnect.  There can be other reasons
for this as well.

In general, any client should be designed with the realization that during a
disconnected period, it is not clear what the situation is.  For instance,
if you are doing leader elect and need to have at least one server running,
then the client should continue serving in whatever roll it was before it
was disconnected.  If, on the other hand, you must avoid having more than
one such server then the client should stop serving in whatever capacity it
is serving in until it either gets a reconnect or an expiration.  If you
want to have your best estimate and minimize time with zero or more than one
server running but put equal weight on both kinds of errors, then the client
should start a timer on disconnect and only stop serving when the client
estimates that the quorum would have marked it as expired.

Clearly, all such strategies are subject to error.  For instance, the timer
approach is subject to error if the clock does not operate the way you
expect or if the entire Zookeeper quorum is down.  These problems are
inherent in distributed systems with fallible communications.  Deal with it.

Most importantly, do not expect that a disconnected client will receive a
session expiration event while it is still disconnected.  These expiration
events come from the server and if you are disconnected, you cannot receive
them.  If and when you reconnect you will receive the right event, but not
until then.


> I have since changed this code to loop trying to check existence of some
> znode upon receipt of a disconnect.  If a session expiration occurs in this
> loop then I trigger the reconnect logic.  Otherwise once we reconnect, the
> check will succeed and the loop will exit.  Does this sound like a
> reasonable way to handle the issue?
>
> Thanks,
> Martin
>
> >
> > Ted,
> >
> > Sorry to trouble you on this one.  I do understand the difference, but at
> > some point I did not.  :)
> >
> > Your question inspired me to look deeper at our code (to see if we were
> > confused) and I found one case that was triggering our reconnect response
> > from Disconnected event.  Everywhere else we only do this in response to
> a
> > SessionExpiredException.
> >
> > Thanks for the quick response and your work on ZooKeeper in general!  I
> > have also run into the "can't create ephemeral yet case" and our code
> > generally loops until successful.
> >
> > -Martin
>
> > > -----Original Message-----
> > > From: Ted Dunning [mailto:ted.dunning@gmail.com]
> > >
> > > Martin,
> > >
> > > From your email, it sounds like there might be a bit of confusion
> > > between disconnection and session expiration.  Are you sure you are
> > > clear on the difference between these?
> > >
> > > Also, I have seen cases in my own code where I confused myself by
> > > trying to re-create ephemeral files after a client program crashed.  I
> > > knew that the client had crashed as soon as it happened, but the
> > > Zookeeper servers could only determine this after a bit of time.  My
> > > new program tried to recreate the ephemerals to indicate that it was
> > > back but since the old ephemerals were still there, that failed.  Then
> > > a short time later when the ZK cluster understood that the old client
> > > was gone, the ephemerals disappeared even though the new client was
> > > humming along nicely.  My solution was to delete the ephemerals when
> > creating them.
> > >
> > > Is it possible you have a similar confusion?
> > >
> > > On Tue, Sep 13, 2011 at 11:25 AM, Martin Serrano <martin@attivio.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We have added code to our application to reconnect and re-establish
> > > > watches when we receive a Disconnected event.  I am running tests on
> > > > a heavily loaded system where the zookeeper server and clients are
> > > > all impacted.  On this test system we regularly experience session
> > > > timeouts and appropriately react to reconnect and set up our watches.
> > > > There is an uncommon case that I am having trouble puzzling out.
> > > > When running one of our tests in a loop about 1% of the time we hit
> > > > a case where
> > > on the client side we think the
> > > > session has expired but on the server side it has been renewed.   We
> will
> > > > then fail to be able to create an ephemeral node because it already
> > > > exists and does not ever get cleaned up (since the previous session
> > > > is still valid).  I'm trying to figure out if we are misusing the API
> or if we
> > have
> > > > encountered a bug.   I'm happy to provide more details.  One thing I
> am
> > > > wondering is if it is inappropriate to create a new session within
> > > > the event thread of another session which has received the
> > > > disconnected
> > > event.
> > > >
> > > > Thanks,
> > > > Martin Serrano
> > > > ...
>
>

--bcaec51a8276861de604acecdab2--