Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of jared.cantwell@gmail.com
 designates 209.85.160.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BANLkTinrbRTk7vXrjCF3vWUfBZ6i7jAdMw@mail.gmail.com>
References: <15121695A88D46F89D330AC7D0F54B75@china.huawei.com>
 <FFA3BDB6-1C83-42B9-B2A0-7675134626C5@me.com>
 <69D3016305F9084FBD2C4A0DF189BD5C17689336FF@GSCMAMP02EX.firmwide.corp.gs.com>
 <BANLkTimgyQWV-pZnRsG7hYO_RpNLoX4fBQ@mail.gmail.com>
 <BANLkTinmpUDiB=j9TpSLavJW4cq-7kXj9A@mail.gmail.com>
 <929B0DD7-9EA6-499C-A14E-A2346DC089CF@me.com>
 <BANLkTinrbRTk7vXrjCF3vWUfBZ6i7jAdMw@mail.gmail.com>
From: Jared Cantwell <jared.cantwell@gmail.com>
Date: Fri, 1 Jul 2011 09:06:23 -0600
Message-ID: <BANLkTim4s47Vu8O03VvZPK6YLeCq9SxW3g@mail.gmail.com>
Subject: Re: Serious problem processing hearbeat on login stampede
To: user@zookeeper.apache.org
Cc: Chang Song <tru64ufs@me.com>
Content-Type: multipart/alternative; boundary=0016363b8e8c8ccc5e04a7035d9e

--0016363b8e8c8ccc5e04a7035d9e
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

As a note, I believe we just used this patch to solve a major issue we were
seeing.  We were having problems when power to a node was pulled, and thus
hung tcp sessions on the servers.  With many connections, each close
operation was taking 2 seconds and held up the server significantly enough
to start incorrectly closing other sessions.  By disabling linger, these
hanging sessions were closed immediately and the problem went away.

Thanks Chang!
~Jared

On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <ted.dunning@gmail.com> wrote=
:

> Where is this set?
>
> Why does this cause this problem?
>
> 2011/4/19 Chang Song <tru64ufs@me.com>
>
> >
> > Problem solved.
> > it was socket linger option set to 2 sec timeout.
> >
> > We have verified that the original problem goes away when we turn off
> > linger option.
> > No longer a mystery ;)
> >
> >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1049
> >
> >
> > Chang
> >
> >
> > 2011. 4. 19., =EC=98=A4=EC=A0=84 3:16, Mahadev Konar =EC=9E=91=EC=84=B1=
:
> >
> > > Camille, Ted,
> > > Can we continue the discussion on
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
> > >
> > > We should track all the suggestions/issues on the jira.
> > >
> > > thanks
> > > mahadev
> > >
> > > On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <ted.dunning@gmail.com>
> > wrote:
> > >> Interesting.  It does seem to suggestion the session expiration is
> > >> expensive.
> > >>
> > >> There is a concurrent table in guava that provides very good
> > multi-threaded
> > >> performance.  I think that is achieved by using a number of locks an=
d
> > then
> > >> distributing threads across the locks according to the hash slot bei=
ng
> > used.
> > >>  But I would have expected any in memory operation to complete very
> > quickly.
> > >>
> > >> Is it possible that the locks on the session table are held longer
> than
> > they
> > >> should be?
> > >>
> > >> 2011/4/18 Fournier, Camille F. [Tech] <Camille.Fournier@gs.com>
> > >>
> > >>> Is it possible this is related to this report back in February?
> > >>>
> > >>>
> >
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%3C66=
42FC1CAF133548AA8FDF497C547F0A23C0C5265B@NYWEXMBX2126.msad.ms.com%3E
> > >>>
> > >>> I theorized that the issue might be due to synchronization on the
> > session
> > >>> table, but never got enough information to finish the investigation=
.
> > >>>
> > >>
> > >
> > >
> > >
> > > --
> > > thanks
> > > mahadev
> > > @mahadevkonar
> >
> >
>

--0016363b8e8c8ccc5e04a7035d9e--