Return-Path: 
 <user-return-4090-apmail-zookeeper-user-archive=zookeeper.apache.org@zookeeper.apache.org>
X-Original-To: apmail-zookeeper-user-archive@www.apache.org
Delivered-To: apmail-zookeeper-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 64DCF9210
	for <apmail-zookeeper-user-archive@www.apache.org>;
 Thu, 10 Nov 2011 21:52:41 +0000 (UTC)
Received: (qmail 70511 invoked by uid 500); 10 Nov 2011 21:52:40 -0000
Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org
Received: (qmail 70454 invoked by uid 500); 10 Nov 2011 21:52:40 -0000
Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@zookeeper.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@zookeeper.apache.org>
List-Post: <mailto:user@zookeeper.apache.org>
List-Id: <user.zookeeper.apache.org>
Reply-To: user@zookeeper.apache.org
Delivered-To: mailing list user@zookeeper.apache.org
Received: (qmail 70439 invoked by uid 99); 10 Nov 2011 21:52:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Nov 2011 21:52:40 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of neha.narkhede@gmail.com
 designates 209.85.212.42 as permitted sender)
Received: from [209.85.212.42] (HELO mail-vw0-f42.google.com) (209.85.212.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Nov 2011 21:52:34 +0000
Received: by vwl1 with SMTP id 1so3250980vwl.15
        for <multiple recipients>; Thu, 10 Nov 2011 13:52:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        bh=9aMlGoa09vLfhEEbIsTjMJaugB6LjHuB+PCg9dJ1iUE=;
        b=u5HeriPHIZnNFOTqxp6IReROBczCX+a8MyRXwljOPPkEwvASZwrED25S0ybE3wIHua
         yPj5w06ydXhEYIuhtLnJcgnAAyx6SdDsgSa2RXSMaf3ycD39BmGi/Ip7fWOybPxrdexq
         0b0expSkroVrx8jTEoB39668YpVUxBb8Rkd/Y=
MIME-Version: 1.0
Received: by 10.52.29.9 with SMTP id f9mr16034392vdh.30.1320961933403; Thu, 10
 Nov 2011 13:52:13 -0800 (PST)
Received: by 10.220.148.130 with HTTP; Thu, 10 Nov 2011 13:52:13 -0800 (PST)
In-Reply-To: 
 <CABWqe2bYGedUKp5cytjdeHHSgfReEo_ReT86XoTKQJD8s_DPeg@mail.gmail.com>
References: 
 <CAOG_4QaL7u4Pt-PKb2L+WdC518UUoToWvdQ2FTxWzn6kE7LAUQ@mail.gmail.com>
	<CANLc_9JYVQrXBjwGOG40V6N1sw1vdTgPPjT0MVgncAiOuGasgg@mail.gmail.com>
	<CANLc_9+OyPos6Lu6woxibmAUz8b1LTN2na3VGMDimjzmigBJew@mail.gmail.com>
	<CANLc_9+y7YN+TRVO_SxObXU8j4OqF7MqpxCRbwvv+N14b0Mi8A@mail.gmail.com>
	<CANLc_9Lth7GfkwQ1evKnR8sVqmr=-b9HTMEWHFcwGy30zzwHPQ@mail.gmail.com>
	<CABWqe2bYGedUKp5cytjdeHHSgfReEo_ReT86XoTKQJD8s_DPeg@mail.gmail.com>
Date: Thu, 10 Nov 2011 13:52:13 -0800
Message-ID: 
 <CAOG_4QamCiphMeuKOcfq-Bkw6KDjmeEBQG-HMFC8roRJw9S2Og@mail.gmail.com>
Subject: Re: ephemeral node not deleted after client session closed
From: Neha Narkhede <neha.narkhede@gmail.com>
To: Camille Fournier <camille@apache.org>
Cc: dev@zookeeper.apache.org, user@zookeeper.apache.org
Content-Type: multipart/alternative; boundary=20cf307cfd4ecd654a04b1686aa7

--20cf307cfd4ecd654a04b1686aa7
Content-Type: text/plain; charset=ISO-8859-1

Thanks for the quick responses, guys! Please find my replies inline -

>> 1) Why is the session closed, the client closed it or the cluster
expired it?
Cluster expired it.

>> 2) which server was the session attached to - the first (44sec max
lat) or one of the others? Which server was the leader?
We didn't run srvr, so no idea which server was the leader. I still have
the zookeeper log4j and txn logs, as well as the client logs. Where do I
look to find this out ?

>> 3) the znode exists on all 4 servers, is that right?
Yes

>> 5) why are your max latencies, as well as avg latency, so high?
>> a) are these dedicated boxes, not virtualized, correct?
these are dedicated boxes, but zk is currently co-located with kafka, but
on different disks

>> b) is the jvm going into gc pause? (try turning on verbose logging, or
use "jstat" with the gc options to see the history if you still have
those jvms running)
I don't believe we had gc logs on these machines. So its unclear.

>> d) do you have dedicated spindles for the ZK WAL? If not another
process might be causing the fsyncs to pause. (you can use iostat or
strace to monitor this)
No. The log4j and zk txn logs share the same disks.

>> Is that the log from the server that's got the 44sec max latency?
Yes.

>> This is 3.3.3 ?
Yes.

>> was there any instability in the quorum itself during this time
period?
How do I find that out ?

Thanks,
Neha


On Thu, Nov 10, 2011 at 1:34 PM, Camille Fournier <camille@apache.org>wrote:

> This is zk 3.3.3?
>
> From my phone
> On Nov 10, 2011 4:02 PM, "Patrick Hunt" <phunt@apache.org> wrote:
>
>> On Thu, Nov 10, 2011 at 12:20 PM, Patrick Hunt <phunt@apache.org> wrote:
>> > Q: what are you clients doing? It's weird that a create would come
>> > from the client after the session has been closed. Do you have
>> > multiple threads sharing a session?
>>
>> The client (checked java) seems to protect against this. Was the
>> session expired? IC. Based on the cxid of the close session being 0
>> I'm guessing it's an expiration.
>>
>> Patrick
>>
>

--20cf307cfd4ecd654a04b1686aa7--