Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6706D9848 for ; Tue, 27 Sep 2011 20:55:20 +0000 (UTC) Received: (qmail 27248 invoked by uid 500); 27 Sep 2011 20:55:19 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 27224 invoked by uid 500); 27 Sep 2011 20:55:19 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 27216 invoked by uid 99); 27 Sep 2011 20:55:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Sep 2011 20:55:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of g.kishore@gmail.com designates 209.85.215.170 as permitted sender) Received: from [209.85.215.170] (HELO mail-ey0-f170.google.com) (209.85.215.170) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Sep 2011 20:55:13 +0000 Received: by eyh6 with SMTP id 6so6137221eyh.15 for ; Tue, 27 Sep 2011 13:54:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=0BeWC6OVXHnfYqLUTzyoUGC022fip7CgrWA0gAaAX8E=; b=EVPMTIMZd/lYTMs2AdPAKdApOjDko01CQ7AOsw1MZ9C0ZRKoe9HHsC7AWaQTJXmtTS kebHh9CNVjNnvulp8SlkEYHP/VVa7+V3BwZngz3NVkcEIiu/km2X9V5jUXKL0bjsgTHc gD66hlhSOUTbLmMcEgL3fOP8aH2aMs6Xz3TrM= MIME-Version: 1.0 Received: by 10.68.34.138 with SMTP id z10mr39148866pbi.105.1317156890811; Tue, 27 Sep 2011 13:54:50 -0700 (PDT) Received: by 10.143.3.1 with HTTP; Tue, 27 Sep 2011 13:54:50 -0700 (PDT) In-Reply-To: References: Date: Tue, 27 Sep 2011 13:54:50 -0700 Message-ID: Subject: Re: ephemeral node not removed after the client session is long gone From: kishore g To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary=bcaec5216303971e8604adf27ca1 --bcaec5216303971e8604adf27ca1 Content-Type: text/plain; charset=ISO-8859-1 Any idea on why this is happened. I tried reproducing this by doing the following repeatedly * create a session * create ephemeral node * close the session Could not reproduce the issue. We can provide additional details if needed thanks, Kishore G On Sun, Sep 25, 2011 at 10:06 PM, kishore g wrote: > Hi, > > I got the following information from the logs. > > The node that still exists is > /kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > > I saw that the ephemeral owner is 86167322861045079 which is session id > 0x13220b93e610550. > > After searching in the transaction log of one of the ZK servers found that > session expired > > 9/22/11 12:17:57 PM PDT session 0x13220b93e610550 cxid 0x74 zxid > 0x601bd36f7 closeSession null > > On digging further into the logs I found that there were multiple sessions > created in quick succession and every session tried to create the same node. > But i verified that the sessions were closed and opened in order > 9/22/11 12:17:56 PM PDT session 0x13220b93e610550 cxid 0x0 zxid 0x601bd36b5 > createSession 6000 > 9/22/11 12:17:57 PM PDT session 0x13220b93e610550 cxid 0x74 zxid > 0x601bd36f7 closeSession null > 9/22/11 12:17:58 PM PDT session 0x13220b93e610551 cxid 0x0 zxid 0x601bd36f8 > createSession 6000 > 9/22/11 12:17:59 PM PDT session 0x13220b93e610551 cxid 0x74 zxid > 0x601bd373a closeSession null > 9/22/11 12:18:00 PM PDT session 0x13220b93e610552 cxid 0x0 zxid 0x601bd373e > createSession 6000 > 9/22/11 12:18:01 PM PDT session 0x13220b93e610552 cxid 0x6c zxid > 0x601bd37a0 closeSession null > 9/22/11 12:18:02 PM PDT session 0x13220b93e610553 cxid 0x0 zxid 0x601bd37e9 > createSession 6000 > 9/22/11 12:18:03 PM PDT session 0x13220b93e610553 cxid 0x74 zxid > 0x601bd382b closeSession null > 9/22/11 12:18:04 PM PDT session 0x13220b93e610554 cxid 0x0 zxid 0x601bd383c > createSession 6000 > 9/22/11 12:18:05 PM PDT session 0x13220b93e610554 cxid 0x6a zxid > 0x601bd388f closeSession null > 9/22/11 12:18:06 PM PDT session 0x13220b93e610555 cxid 0x0 zxid 0x601bd3895 > createSession 6000 > 9/22/11 12:18:07 PM PDT session 0x13220b93e610555 cxid 0x6a zxid > 0x601bd38cd closeSession null > 9/22/11 12:18:10 PM PDT session 0x13220b93e610556 cxid 0x0 zxid 0x601bd38d1 > createSession 6000 > 9/22/11 12:18:11 PM PDT session 0x13220b93e610557 cxid 0x0 zxid 0x601bd38f2 > createSession 6000 > 9/22/11 12:18:11 PM PDT session 0x13220b93e610557 cxid 0x51 zxid > 0x601bd396a closeSession null > > Here is the log output for the sessions that tried creating the same node > > 9/22/11 12:17:54 PM PDT session 0x13220b93e61054f cxid 0x42 zxid > 0x601bd366b create > '/kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > 9/22/11 12:17:56 PM PDT session 0x13220b93e610550 cxid 0x42 zxid > 0x601bd36ce create > '/kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > 9/22/11 12:17:58 PM PDT session 0x13220b93e610551 cxid 0x42 zxid > 0x601bd3711 create > '/kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > 9/22/11 12:18:00 PM PDT session 0x13220b93e610552 cxid 0x42 zxid > 0x601bd3777 create > '/kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > 9/22/11 12:18:02 PM PDT session 0x13220b93e610553 cxid 0x42 zxid > 0x601bd3802 create > '/kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > 9/22/11 12:18:05 PM PDT session 0x13220b93e610554 cxid 0x44 zxid > 0x601bd385d create > '/kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > 9/22/11 12:18:07 PM PDT session 0x13220b93e610555 cxid 0x44 zxid > 0x601bd38b0 create > '/kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > 9/22/11 12:18:11 PM PDT session 0x13220b93e610557 cxid 0x52 zxid > 0x601bd396b create > '/kafka-tracking/consumers/UserPerformanceEvent-/owners/UserPerformanceEvent/529-7 > > Let me know if you need additional information. > > Thanks, > Kishore G > > > On Sun, Sep 25, 2011 at 12:56 PM, Camille Fournier wrote: > >> Can you give us more details, like the information on the client that >> created it, any logs around the time they created it/disconnected, >> etc? >> >> On Sun, Sep 25, 2011 at 2:54 PM, Jun Rao wrote: >> > Hi, >> > >> > We found our ZK server in a state where an ephemeral node still exists >> after >> > a client session is long gone. I used the cons command on each ZK host >> to >> > list all connections and couldn't find the ephemeralOwner id. We are >> using >> > ZK 3.3.3. Has anyone seen this problem? >> > >> > Thanks, >> > >> > Jun >> > >> > > --bcaec5216303971e8604adf27ca1--