hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: closing session on socket close vs waiting for timeout
Date Tue, 07 Sep 2010 17:10:31 GMT
This really is, just as Ben says a problem of false positives and false
negatives in detecting session
expiration.

On the other hand, the current algorithm isn't really using all the
information available.  The current algorithm is
using time since last client initiated heartbeat.  The new proposal is
somewhat worse in that it proposes to use
just the boolean "has-TCP-disconnect-happened".

Perhaps it would be better to use multiple features in order to decrease
both false positives and false negatives.

For instance, I could imagine that we use the following features:

- time since last client hearbeat or disconnect or reconnect

- what was the last event? (a heartbeat or a disconnect or a reconnect)

Then the expiration algorithm could use a relatively long time since last
heartbeat and a relatively short time since last disconnect to mark a
session as disconnected.

Wouldn't this avoid expiration during GC and cluster partition and cause
expiration quickly after a client disconnect?


On Mon, Sep 6, 2010 at 11:26 PM, Patrick Hunt <phunt@apache.org> wrote:

> That's a good point, however with suitable documentation, warnings and such
> it seems like a reasonable feature to provide for those users who require
> it. Used in moderation it seems fine to me. Perhaps we also make it
> configurable at the server level for those administrators/ops who don't
> want
> to deal with it (disable the feature entirely, or only enable on particular
> servers, etc...).
>
> Patrick
>
> On Mon, Sep 6, 2010 at 2:10 PM, Benjamin Reed <breed@yahoo-inc.com> wrote:
>
> > if this mechanism were used very often, we would get a huge number of
> > session expirations when a server fails. you are trading fast error
> > detection for the ability to tolerate temporary network and server
> outages.
> >
> > to be honest this seems like something that in theory sounds like it will
> > work in practice, but once deployed we start getting session expirations
> for
> > cases that we really do not want or expect.
> >
> > ben
> >
> >
> > On 09/01/2010 12:47 PM, Patrick Hunt wrote:
> >
> >> Ben, in this case the session would be tied directly to the connection,
> >> we'd explicitly deny session re-establishment for this session type (so
> >> 4 would fail). Would that address your concern, others?
> >>
> >> Patrick
> >>
> >> On 09/01/2010 10:03 AM, Benjamin Reed wrote:
> >>
> >>
> >>> i'm a bit skeptical that this is going to work out properly. a server
> >>> may receive a socket reset even though the client is still alive:
> >>>
> >>> 1) client sends a request to a server
> >>> 2) client is partitioned from the server
> >>> 3) server starts trying to send response
> >>> 4) client reconnects to a different server
> >>> 5) partition heals
> >>> 6) server gets a reset from client
> >>>
> >>> at step 6 i don't think you want to delete the ephemeral nodes.
> >>>
> >>> ben
> >>>
> >>> On 08/31/2010 01:41 PM, Fournier, Camille F. [Tech] wrote:
> >>>
> >>>
> >>>> Yes that's right. Which network issues can cause the socket to close
> >>>> without the initiating process closing the socket? In my limited
> >>>> experience in this area network issues were more prone to leave dead
> >>>> sockets open rather than vice versa so I don't know what to look out
> >>>> for.
> >>>>
> >>>> Thanks,
> >>>> Camille
> >>>>
> >>>> -----Original Message-----
> >>>> From: Dave Wright [mailto:wrightd@gmail.com]
> >>>> Sent: Tuesday, August 31, 2010 1:14 PM
> >>>> To: zookeeper-user@hadoop.apache.org
> >>>> Subject: Re: closing session on socket close vs waiting for timeout
> >>>>
> >>>> I think he's saying that if the socket closes because of a crash (i.e.
> >>>> not a
> >>>> normal zookeeper close request) then the session stays alive until the
> >>>> session timeout, which is of course true since ZK allows reconnection
> >>>> and
> >>>> resumption of the session in case of disconnect due to network issues.
> >>>>
> >>>> -Dave Wright
> >>>>
> >>>> On Tue, Aug 31, 2010 at 1:03 PM, Ted Dunning<ted.dunning@gmail.com>
> >>>> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> That doesn't sound right to me.
> >>>>>
> >>>>> Is there a Zookeeper expert in the house?
> >>>>>
> >>>>> On Tue, Aug 31, 2010 at 8:58 AM, Fournier, Camille F. [Tech]<
> >>>>> Camille.Fournier@gs.com>  wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> I foolishly did not investigate the ZK code closely enough and
it
> >>>>>> seems
> >>>>>> that closing the socket still waits for the session timeout
to
> >>>>>> remove the
> >>>>>> session.
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message