zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jared Cantwell <jared.cantw...@gmail.com>
Subject Re: Serious problem processing hearbeat on login stampede
Date Fri, 01 Jul 2011 15:06:23 GMT
As a note, I believe we just used this patch to solve a major issue we were
seeing.  We were having problems when power to a node was pulled, and thus
hung tcp sessions on the servers.  With many connections, each close
operation was taking 2 seconds and held up the server significantly enough
to start incorrectly closing other sessions.  By disabling linger, these
hanging sessions were closed immediately and the problem went away.

Thanks Chang!
~Jared

On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Where is this set?
>
> Why does this cause this problem?
>
> 2011/4/19 Chang Song <tru64ufs@me.com>
>
> >
> > Problem solved.
> > it was socket linger option set to 2 sec timeout.
> >
> > We have verified that the original problem goes away when we turn off
> > linger option.
> > No longer a mystery ;)
> >
> >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1049
> >
> >
> > Chang
> >
> >
> > 2011. 4. 19., 오전 3:16, Mahadev Konar 작성:
> >
> > > Camille, Ted,
> > > Can we continue the discussion on
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
> > >
> > > We should track all the suggestions/issues on the jira.
> > >
> > > thanks
> > > mahadev
> > >
> > > On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <ted.dunning@gmail.com>
> > wrote:
> > >> Interesting.  It does seem to suggestion the session expiration is
> > >> expensive.
> > >>
> > >> There is a concurrent table in guava that provides very good
> > multi-threaded
> > >> performance.  I think that is achieved by using a number of locks and
> > then
> > >> distributing threads across the locks according to the hash slot being
> > used.
> > >>  But I would have expected any in memory operation to complete very
> > quickly.
> > >>
> > >> Is it possible that the locks on the session table are held longer
> than
> > they
> > >> should be?
> > >>
> > >> 2011/4/18 Fournier, Camille F. [Tech] <Camille.Fournier@gs.com>
> > >>
> > >>> Is it possible this is related to this report back in February?
> > >>>
> > >>>
> >
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%3C6642FC1CAF133548AA8FDF497C547F0A23C0C5265B@NYWEXMBX2126.msad.ms.com%3E
> > >>>
> > >>> I theorized that the issue might be due to synchronization on the
> > session
> > >>> table, but never got enough information to finish the investigation.
> > >>>
> > >>
> > >
> > >
> > >
> > > --
> > > thanks
> > > mahadev
> > > @mahadevkonar
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message