Return-Path: 
 <user-return-596-apmail-curator-user-archive=curator.apache.org@curator.apache.org>
X-Original-To: apmail-curator-user-archive@minotaur.apache.org
Delivered-To: apmail-curator-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 35E8A10B0C
	for <apmail-curator-user-archive@minotaur.apache.org>;
 Thu, 27 Mar 2014 17:31:20 +0000 (UTC)
Received: (qmail 31401 invoked by uid 500); 27 Mar 2014 17:31:17 -0000
Delivered-To: apmail-curator-user-archive@curator.apache.org
Received: (qmail 31359 invoked by uid 500); 27 Mar 2014 17:31:16 -0000
Mailing-List: contact user-help@curator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@curator.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@curator.apache.org>
List-Post: <mailto:user@curator.apache.org>
List-Id: <user.curator.apache.org>
Reply-To: user@curator.apache.org
Delivered-To: mailing list user@curator.apache.org
Received: (qmail 31329 invoked by uid 99); 27 Mar 2014 17:31:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Mar 2014 17:31:15 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
Received: from [209.85.214.170] (HELO mail-ob0-f170.google.com)
 (209.85.214.170)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Mar 2014 17:31:10 +0000
Received: by mail-ob0-f170.google.com with SMTP id uz6so4701685obc.29
        for <user@curator.apache.org>; Thu, 27 Mar 2014 10:30:49 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:date:from:to:message-id:in-reply-to:references
         :subject:mime-version:content-type;
        bh=1CQtrFY7xrSS2Qn70O5bdA3pKWF7TDlrJwXOW1NGLEE=;
        b=AvFpAi2rF8XxMmhywXzFAqhGSbqXlHhLqd/4GjDhNgRuSieV7ssJTzgGUuwLDz1eCX
         4UAgDRi973jcb4ZvsisUaZQMXOiEGE7fwqZPxa11tC3vsACgcwV3ylsVOLlmzmFUZ68D
         8g79HzNKqzPkG34HiUapphBPW4/kahMEWL7Icqm5cHGhT5iIuK8AT5OBq9wajkA0uVDo
         AItBkDhgQMutYO4irpYStNRCFqBbAH6GybikglxLDMgraLYAddMOJs/xQCLN097I88nf
         WO44VI05OauLsdEat5gbWd+Pj1FFMyNCKOgUfvrjoB29VObjdowYHuaokZcnx/DhgqZq
         lPlw==
X-Gm-Message-State: 
 ALoCoQnvsiXgL1/N4DG7uQF46z+2h1l5kKjvEheeEFZoaMPc74adKQK9ivlH1bmexiaCZOdTUVJh
X-Received: by 10.182.33.6 with SMTP id n6mr2497187obi.48.1395941448960;
        Thu, 27 Mar 2014 10:30:48 -0700 (PDT)
Received: from Jordans-MacBook-Pro.local (cpe-907240009601.cpe.cableonda.net.
 [190.219.168.142])
        by mx.google.com with ESMTPSA id m7sm4338397obo.7.2014.03.27.10.30.46
        for <multiple recipients>
        (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
        Thu, 27 Mar 2014 10:30:48 -0700 (PDT)
Date: Thu, 27 Mar 2014 12:30:26 -0500
From: Jordan Zimmerman <jordan@jordanzimmerman.com>
To: Brian Phillips <brian@etinternational.com>, user@curator.apache.org
Message-ID: <etPan.53346042.354fe9f9.14f7@Jordans-MacBook-Pro.local>
In-Reply-To: <20140327172552.e827b205@mail02.etinternational.com>
References: <20140327172552.e827b205@mail02.etinternational.com>
Subject: Re: Curator barriers missing watch events
X-Mailer: Airmail (231)
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="53346042_15b5af5c_14f7"
X-Virus-Checked: Checked by ClamAV on apache.org

--53346042_15b5af5c_14f7
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

https://cwiki.apache.org/confluence/display/CURATOR/TN1

:)


=46rom:=C2=A0Brian Phillips brian=40etinternational.com
Reply:=C2=A0user=40curator.apache.org user=40curator.apache.org
Date:=C2=A0March 27, 2014 at 12:26:32 PM
To:=C2=A0user=40curator.apache.org user=40curator.apache.org
Subject:=C2=A0 Re: Curator barriers missing watch events =20

I finally figured out my problem, and it was my fault. Hopefully someone =
else can learn from this.

What was happening was that I was using a zookeeper watch event to kick o=
ff a bunch of code, which then ended up in the zookeeper/curator barrier.=
 Since the watch thread was the one that executed the barrier, it blocked=
 itself from receiving any additional watch events from zookeeper, includ=
ing the ones that the barrier depended upon.

So as a general rule, DON'T BLOCK YOUR WATCH THREADS. I feel stupid for n=
ot realizing this sooner.

=5FBrian=3D
=46rom: Brian Phillips =5Bmailto:brian=40etinternational.com=5D
To: user=40curator.apache.org
Sent: Wed, 26 Mar 2014 15:32:13 -0500
Subject: Re: Curator barriers missing watch events

So I'm still working on this issue. I grabbed a zookeeper only barrier im=
plementation from here:

http://zookeeper.apache.org/doc/r3.3.3/zookeeperTutorial.html

This barrier makes it's own zookeeper connection=C2=A0separately=C2=A0fro=
m the curator connection that my program uses. When I put this barrier in=
to my program, everything works as it should, and nobody gets stuck on th=
e barriers. I then modified the barrier to use curators connection, passi=
ng in=C2=A0Curator=46ramework.getZookeeperClient().getZooKeeper() instead=
 of connecting separately. Once I did this, it breaks exactly as it did b=
efore when using the curator barrier.

This seems to indicate to me that something else I've done in the program=
 has 'broken' the zookeeper session associated with my curator connection=
, to the point where some watch events no longer work.

I'm going to embark on the arduous process of trying to figure out what I=
'm doing thats breaking my sessions watches. Watches not working properly=
 is disturbing, and will certainly prevent other parts of my program from=
 functioning correctly, probably in less obvious ways.

=5FBrian=3D
=46rom: Brian Phillips =5Bmailto:brian=40etinternational.com=5D
To: user=40curator.apache.org =5Bmailto:user=40curator.apache.org=5D
Sent: Tue, 25 Mar 2014 20:39:31 -0500
Subject: Re: Curator barriers missing watch events

Yes, there's two barrier sessions. But different barrier instances, and d=
ifferent barrier paths. ):

Sent from my iPhone

On Mar 25, 2014, at 8:34 PM, =22Jordan Zimmerman=22 <jordan=40jordanzimme=
rman.com> wrote:

Are you saying there are two barrier sessions=3F The first one works, but=
 the second doesn=E2=80=99t=3F Are you re-using the same path=3F I wonder=
 if there are znodes left in the path or something. Before running the se=
cond barrier session, double check that the path is empty (do a getChildr=
en on it). If it=E2=80=99s not empty that could be the problem.

-JZ


=46rom:=C2=A0Brian Phillips brian=40etinternational.com
Reply:=C2=A0user=40curator.apache.org user=40curator.apache.org
Date:=C2=A0March 25, 2014 at 6:10:46 PM
To:=C2=A0user=40curator.apache.org user=40curator.apache.org
Subject:=C2=A0 Re: Curator barriers missing watch events

I=E2=80=99ve tried, but it seems to be timing specific. Its in a rather l=
arge complicated program, where the first barrier always works but the on=
e at the end of the program usually gets stuck. I=E2=80=99ve spent all da=
y trying to make sense of it, as my project really needs it to work.

I=E2=80=99d like to be able to figure out if the zookeeper server is actu=
ally sending my clients the watch events.=C2=A0

=5FB


On Mar 25, 2014, at 6:53 PM, =22Jordan Zimmerman=22 <jordan=40jordanzimme=
rman.com> wrote:

There=E2=80=99s no way you can distill your usage into a test=3F

-JZ


=46rom:=C2=A0Brian Phillips brian=40etinternational.com
Reply:=C2=A0user=40curator.apache.org user=40curator.apache.org
Date:=C2=A0March 25, 2014 at 5:51:37 PM
To:=C2=A0user=40curator.apache.org user=40curator.apache.org
Subject:=C2=A0 Re: Curator barriers missing watch events

Hmm, I made that change, but it didn't seem to help. The first program ma=
de it to the barrier enter, then the second program entered, exited, and =
the first program never left the barrier.

The second program got a node created event, but the first program never =
got any event from its watcher.

I appreciate the help=21 Must be something else.

=5FB

On Mar 25, 2014, at 6:28 PM, =22Jordan Zimmerman=22 <jordan=40jordanzimme=
rman.com> wrote:

Look at line 313 and line 331. The noarg version of enter() causes=C2=A0i=
nternalEnter() to call wait even though the watcher may have already noti=
fied. I believe line 331 should be:

else=C2=A0if ( =21hasBeenNotified.get() )

-JZ


=46rom:=C2=A0Brian Phillips brian=40etinternational.com
Reply:=C2=A0user=40curator.apache.org user=40curator.apache.org
Date:=C2=A0March 25, 2014 at 5:25:48 PM
To:=C2=A0user=40curator.apache.org user=40curator.apache.org
Subject:=C2=A0 Re: Curator barriers missing watch events

I am using the no arg version=21 What's the bug=3F

=5FB

On Mar 25, 2014, at 6:23 PM, =22Jordan Zimmerman=22 <jordan=40jordanzimme=
rman.com> wrote:

Which version of enter() are you using=3F I see a potential bug when the =
no arg version of enter() is used.


=46rom:=C2=A0Brian Phillips brian=40etinternational.com
Reply:=C2=A0Brian Phillips brian=40etinternational.com
Date:=C2=A0March 25, 2014 at 4:19:36 PM
To:=C2=A0Jordan Zimmerman jordan=40jordanzimmerman.com
Subject:=C2=A0 Re: Curator barriers missing watch events

Good idea, but yes I am. The connection state doesn=E2=80=99t change whil=
e I=E2=80=99m executing the barrier code. It seems to be some kind of rac=
e condition I think, as sometimes it work and sometimes it doesn=E2=80=99=
t. I=E2=80=99ve looked through the recipe code and it looks good as far a=
s I can tell though. I=E2=80=99m practically pulling my hair out at this =
point.

I may try a non-curator zookeeper only barrier tomorrow. See if that work=
s. Or I may start trying to debug the zookeeper client, see if its actual=
ly getting the watches but not delivering them.

=5FB

On Mar 25, 2014, at 4:54 PM, Jordan Zimmerman <jordan=40jordanzimmerman.c=
om> wrote:

Are you setting a ConnectionStateListener=3F If the connection gets SUSPE=
NDED or LOST then you=E2=80=99d need to reinitialize your barrier.

-JZ


=46rom:=C2=A0Brian Phillips=C2=A0brian=40etinternational.com
Reply:=C2=A0user=40curator.apache.org=C2=A0user=40curator.apache.org
Date:=C2=A0March 25, 2014 at 2:51:42 PM
To:=C2=A0user=40curator.apache.org=C2=A0user=40curator.apache.org
Subject:=C2=A0=C2=A0Re: Curator barriers missing watch events=C2=A0

I have tried writing a test program which launches two programs in the sa=
me manor, each makes a connection then loops over barriers with a Thread.=
sleep(random) in-between. This run indefinitely and everything works out =
fine.

I have also tried writing my own barrier, which uses a SharedCount, where=
 each guy tries to increment it until it hits a memberQty. This too misse=
d watch events and does not work properly.

It=E2=80=99s almost as if something else that I=E2=80=99ve done during th=
e running of my program has broken zookeepers watch events somehow. Is th=
ere any good way to debug watch events in general=3F I=E2=80=99ve tried t=
o look at the DEBUG output for my zookeeper server log, but it looks the =
same for the working vs non-working barriers...

=5FB

On Mar 25, 2014, at 3:42 PM, Jordan Zimmerman <jordan=40jordanzimmerman.c=
om> wrote:

Unfortunately, the barrier recipes aren=E2=80=99t widely used (from what =
I know). So, there may well be a bug. If you could get a test to show the=
 problem that would be ideal.

-JZ


=46rom:=C2=A0Brian Phillips=C2=A0brian=40etinternational.com
Reply:=C2=A0user=40curator.apache.org=C2=A0user=40curator.apache.org
Date:=C2=A0March 25, 2014 at 2:38:40 PM
To:=C2=A0user=40curator.apache.org=C2=A0user=40curator.apache.org
Subject:=C2=A0=C2=A0Curator barriers missing watch events=C2=A0

Hi guys,=C2=A0

I=E2=80=99ve been integrating curator into my project and have recently r=
un into an issue I just can=E2=80=99t seem to make sense of.=C2=A0

I=E2=80=99m running two JVMs on the same host machine, each with their ow=
n curator connection. At the beginning of my program I=E2=80=99m using th=
e DistributedDoubleBarrier recipe, and once again at the end of my progra=
m. A bunch of work is done in-between, including zookeeper set/get/watche=
s of other nodes.=C2=A0

I=E2=80=99m finding that the first double barrier, everyone always making=
 it through. The job-end barrier, sometimes everyone gets through, but mo=
re often than not one of the programs hangs in enter's wait(), and never =
gets the watch event for the ready path which notifies it to proceed. If =
I look in zookeeper, I can see that the ready path is actually set in the=
re.=C2=A0

It would seem that the watch for one of the programs just never triggers.=
=C2=A0

To simplify debugging, I=E2=80=99ve set both double barriers to only ever=
 call enter() and not leave(). Both barriers have their own separate path=
.=C2=A0

Also, the program never shuts down or disconnects from zookeeper. It just=
 sleeps infinitely after it gets out of the final barrier.=C2=A0

Any idea on how to debug this issue=3F I don=E2=80=99t mind hacking up zo=
okeeper/curator code to insert my own debugging statements if it comes to=
 that.=C2=A0

=5FBrian=3D


--53346042_15b5af5c_14f7
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

<html><head><style>body=7Bfont-family:Helvetica,Arial;font-size:13px=7D</=
style></head><body style=3D=22word-wrap: break-word; -webkit-nbsp-mode: s=
pace; -webkit-line-break: after-white-space;=22><div id=3D=22bloop=5Fcust=
omfont=22 style=3D=22font-family:Helvetica,Arial;font-size:13px; color: r=
gba(0,0,0,1.0); margin: 0px; line-height: auto;=22><a href=3D=22https://c=
wiki.apache.org/confluence/display/CURATOR/TN1=22>https://cwiki.apache.or=
g/confluence/display/CURATOR/TN1</a></div><div id=3D=22bloop=5Fcustomfont=
=22 style=3D=22font-family:Helvetica,Arial;font-size:13px; color: rgba(0,=
0,0,1.0); margin: 0px; line-height: auto;=22><br></div><div id=3D=22bloop=
=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial;font-size:13px; c=
olor: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22>:)</div> <div i=
d=3D=22bloop=5Fsign=5F1395941419589803008=22 class=3D=22bloop=5Fsign=22><=
div style=3D=22font-family:helvetica,arial;font-size:13px=22><br></div></=
div> <div style=3D=22color:black=22><br>=46rom:&nbsp;<span style=3D=22col=
or:black=22>Brian Phillips</span> <a href=3D=22mailto:brian=40etinternati=
onal.com=22>brian=40etinternational.com</a><br>Reply:&nbsp;<span style=3D=
=22color:black=22>user=40curator.apache.org</span> <a href=3D=22mailto:us=
er=40curator.apache.org=22>user=40curator.apache.org</a><br>Date:&nbsp;<s=
pan style=3D=22color:black=22>March 27, 2014 at 12:26:32 PM</span><br>To:=
&nbsp;<span style=3D=22color:black=22>user=40curator.apache.org</span> <a=
 href=3D=22mailto:user=40curator.apache.org=22>user=40curator.apache.org<=
/a><br>Subject:&nbsp;<span style=3D=22color:black=22> Re: Curator barrier=
s missing watch events <br></span></div><br> <blockquote type=3D=22cite=22=
 class=3D=22clean=5Fbq=22><span><div><div></div><div>






<title></title>


I finally figured out my problem, and it was my fault. Hopefully
someone else can learn from this.
<div><br></div>
<div>What was happening was that I was using a zookeeper watch
event to kick off a bunch of code, which then ended up in the
zookeeper/curator barrier. Since the watch thread was the one that
executed the barrier, it blocked itself from receiving any
additional watch events from zookeeper, including the ones that the
barrier depended upon.</div>
<div><br></div>
<div>So as a general rule, DON'T BLOCK YOUR WATCH THREADS. I feel
stupid for not realizing this sooner.</div>
<div><br></div>
<div>=5FBrian=3D<br>
<blockquote style=3D=22padding-left: 5px; margin-left: 5px; border-left: =
=230000ff 2px solid; margin-right: 0px=22>
<hr>
<b>=46rom:</b> Brian Phillips
=5Bmailto:brian=40etinternational.com=5D<br>
<b>To:</b> user=40curator.apache.org<br>
<b>Sent:</b> Wed, 26 Mar 2014 15:32:13 -0500<br>
<b>Subject:</b> Re: Curator barriers missing watch events<br>
<br>
<font face=3D=22Tahoma, Verdana, sans-serif=22 size=3D=222=22>So I'm stil=
l
working on this issue. I grabbed a zookeeper only barrier
implementation from here:</font>
<div style=3D=22font-family: Tahoma, Verdana, sans-serif; font-size: 10pt=
; font-style: normal; font-variant: normal; font-weight: normal; line-hei=
ght: normal;=22>
<br></div>
<div><a href=3D=22http://zookeeper.apache.org/doc/r3.3.3/zookeeperTutoria=
l.html=22 target=3D=22=5Fblank=22><font face=3D=22Tahoma, Verdana, sans-s=
erif=22>http://zookeeper.apache.org/doc/r3.3.3/zookeeperTutorial.html</fo=
nt></a></div>
<div><font face=3D=22Tahoma, Verdana, sans-serif=22><br></font></div>
<div><font face=3D=22Tahoma, Verdana, sans-serif=22>This barrier makes
it's own zookeeper connection&nbsp;separately&nbsp;from the curator
connection that my program uses. When I put this barrier into my
program, everything works as it should, and nobody gets stuck on
the barriers. I then modified the barrier to use curators
connection, passing
in&nbsp;</font>Curator=46ramework.getZookeeperClient().getZooKeeper()
instead of connecting separately. Once I did this, it breaks
exactly as it did before when using the curator barrier.</div>
<div><br></div>
<div>This seems to indicate to me that something else I've done in
the program has 'broken' the zookeeper session associated with my
curator connection, to the point where some watch events no longer
work.</div>
<div><br></div>
<div>I'm going to embark on the arduous process of trying to figure
out what I'm doing thats breaking my sessions watches. Watches not
working properly is disturbing, and will certainly prevent other
parts of my program from functioning correctly, probably in less
obvious ways.</div>
<div><br></div>
<div>=5FBrian=3D</div>
<div>
<blockquote style=3D=22font-family: Tahoma, Verdana, sans-serif; font-siz=
e: 10pt; font-style: normal; font-variant: normal; font-weight: normal; l=
ine-height: normal; padding-left: 5px; margin-left: 5px; border-left-colo=
r: rgb(0, 0, 255); border-left-width: 2px; border-left-style: solid; marg=
in-right: 0px;=22>
<hr>
<b>=46rom:</b> Brian Phillips =5Bmailto:<a href=3D=22mailto:brian=40etint=
ernational.com=22>brian=40etinternational.com</a>=5D<br>

<b>To:</b> <a href=3D=22mailto:user=40curator.apache.org=22>user=40curato=
r.apache.org</a>
=5Bmailto:<a href=3D=22mailto:user=40curator.apache.org=22>user=40curator=
.apache.org</a>=5D<br>
<b>Sent:</b> Tue, 25 Mar 2014 20:39:31 -0500<br>
<b>Subject:</b> Re: Curator barriers missing watch events<br>
<br>
<div>Yes, there's two barrier sessions. But different barrier
instances, and different barrier paths. ):<br>
<br>
Sent from my iPhone</div>
<div><br>
On Mar 25, 2014, at 8:34 PM, =22Jordan Zimmerman=22 &lt;<a href=3D=22mail=
to:jordan=40jordanzimmerman.com=22>jordan=40jordanzimmerman.com</a>&gt;
wrote:<br>
<br></div>
<blockquote type=3D=22cite=22>
<div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
Are you saying there are two barrier sessions=3F The first one works,
but the second doesn=E2=80=99t=3F Are you re-using the same path=3F I won=
der if
there are znodes left in the path or something. Before running the
second barrier session, double check that the path is empty (do a
getChildren on it). If it=E2=80=99s not empty that could be the
problem.</div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
<br></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
-JZ</div>
<div id=3D=22bloop=5Fsign=5F1395793969289364992=22 class=3D=22bloop=5Fsig=
n=22>
<div style=3D=22font-family:helvetica,arial;font-size:13px=22>
<br></div>
</div>
<div style=3D=22color:black=22><br>
=46rom:&nbsp;<span style=3D=22color:black=22>Brian Phillips</span> <a hre=
f=3D=22mailto:brian=40etinternational.com=22>brian=40etinternational.com<=
/a><br>

Reply:&nbsp;<span style=3D=22color:black=22><a href=3D=22mailto:user=40cu=
rator.apache.org=22>user=40curator.apache.org</a></span>
<a href=3D=22mailto:user=40curator.apache.org=22>user=40curator.apache.or=
g</a><br>
Date:&nbsp;<span style=3D=22color:black=22>March 25, 2014 at 6:10:46
PM</span><br>
To:&nbsp;<span style=3D=22color:black=22><a href=3D=22mailto:user=40curat=
or.apache.org=22>user=40curator.apache.org</a></span>
<a href=3D=22mailto:user=40curator.apache.org=22>user=40curator.apache.or=
g</a><br>
Subject:&nbsp; <span style=3D=22color:black=22>Re: Curator barriers
missing watch events<br></span></div>
<br>
<blockquote type=3D=22cite=22 class=3D=22clean=5Fbq=22>
<div>
<div>
<div>
<p class=3D=22=22><span><span style=3D=22-webkit-text-size-adjust: auto; =
background-color: rgba(255, 255, 255, 0);=22>
I=E2=80=99ve tried, but it seems to be timing specific. Its in a rather
large complicated program, where the first barrier always works but
the one at the end of the program usually gets stuck. I=E2=80=99ve spent
all day trying to make sense of it, as my project really needs it
to work.</span></span></p>
<p class=3D=22=22><span style=3D=22-webkit-text-size-adjust: auto; backgr=
ound-color: rgba(255, 255, 255, 0);=22>
I=E2=80=99d like to be able to figure out if the zookeeper server is
actually sending my clients the watch events.&nbsp;</span></p>
<p class=3D=22=22><span style=3D=22-webkit-text-size-adjust: auto; backgr=
ound-color: rgba(255, 255, 255, 0);=22>
=5FB</span></p>
</div>
<div style=3D=22-webkit-text-size-adjust: auto;=22><br>
On Mar 25, 2014, at 6:53 PM, =22Jordan Zimmerman=22 &lt;<a href=3D=22mail=
to:jordan=40jordanzimmerman.com=22>jordan=40jordanzimmerman.com</a>&gt;
wrote:<br>
<br></div>
<blockquote type=3D=22cite=22 style=3D=22-webkit-text-size-adjust: auto;=22=
>
<div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
There=E2=80=99s no way you can distill your usage into a test=3F</div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
<br></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
-JZ</div>
<div id=3D=22bloop=5Fsign=5F1395787921279940096=22 class=3D=22bloop=5Fsig=
n=22>
<div style=3D=22font-family:helvetica,arial;font-size:13px=22>
<br></div>
</div>
<div style=3D=22color:black=22><br>
=46rom:&nbsp;<span style=3D=22color:black=22>Brian Phillips</span> <a hre=
f=3D=22mailto:brian=40etinternational.com=22>brian=40etinternational.com<=
/a><br>

Reply:&nbsp;<span style=3D=22color:black=22><a href=3D=22mailto:user=40cu=
rator.apache.org=22>user=40curator.apache.org</a></span>
<a href=3D=22mailto:user=40curator.apache.org=22>user=40curator.apache.or=
g</a><br>
Date:&nbsp;<span style=3D=22color:black=22>March 25, 2014 at 5:51:37
PM</span><br>
To:&nbsp;<span style=3D=22color:black=22><a href=3D=22mailto:user=40curat=
or.apache.org=22>user=40curator.apache.org</a></span>
<a href=3D=22mailto:user=40curator.apache.org=22>user=40curator.apache.or=
g</a><br>
Subject:&nbsp; <span style=3D=22color:black=22>Re: Curator barriers
missing watch events<br></span></div>
<br>
<blockquote type=3D=22cite=22 class=3D=22clean=5Fbq=22>
<div>
<div>
<div><span>Hmm, I made that change, but it didn't seem to help. The
first program made it to the barrier enter, then the second program
entered, exited, and the first program never left the
barrier.</span></div>
<div><span><br></span></div>
<div><span>The second program got a node created event, but the
first program never got any event from its watcher.</span></div>
<div><span><br></span></div>
<div><span>I appreciate the help=21 Must be something
else.</span></div>
<div><span><br>
=5FB</span></div>
<div><span><br>
On Mar 25, 2014, at 6:28 PM, =22Jordan Zimmerman=22 &lt;<a href=3D=22mail=
to:jordan=40jordanzimmerman.com=22>jordan=40jordanzimmerman.com</a>&gt;
wrote:<br>
<br></span></div>
<blockquote type=3D=22cite=22>
<div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
<span>Look at line 313 and line 331. The noarg version of enter()
causes&nbsp;internalEnter() to call wait even though the watcher
may have already notified. I believe line 331 should
be:</span></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
<span><br></span></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
<span>else&nbsp;if ( =21hasBeenNotified.get() )</span></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
<span><br></span></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
<span>-JZ</span></div>
<div id=3D=22bloop=5Fsign=5F1395786381560428032=22 class=3D=22bloop=5Fsig=
n=22>
<div style=3D=22font-family:helvetica,arial;font-size:13px=22>
<span><br></span></div>
</div>
<div style=3D=22color:black=22><span><br>
=46rom:&nbsp;<span style=3D=22color:black=22>Brian Phillips</span> <a hre=
f=3D=22mailto:brian=40etinternational.com=22>brian=40etinternational.com<=
/a><br>

Reply:&nbsp;<span style=3D=22color:black=22><a href=3D=22mailto:user=40cu=
rator.apache.org=22>user=40curator.apache.org</a></span>
<a href=3D=22mailto:user=40curator.apache.org=22>user=40curator.apache.or=
g</a><br>
Date:&nbsp;<span style=3D=22color:black=22>March 25, 2014 at 5:25:48
PM</span><br>
To:&nbsp;<span style=3D=22color:black=22><a href=3D=22mailto:user=40curat=
or.apache.org=22>user=40curator.apache.org</a></span>
<a href=3D=22mailto:user=40curator.apache.org=22>user=40curator.apache.or=
g</a><br>
Subject:&nbsp; <span style=3D=22color:black=22>Re: Curator barriers
missing watch events<br></span></span></div>
<br>
<blockquote type=3D=22cite=22 class=3D=22clean=5Fbq=22>
<div>
<div>
<div><span>I am using the no arg version=21 What's the bug=3F<br>
<br>
=5FB</span></div>
<div><span><br>
On Mar 25, 2014, at 6:23 PM, =22Jordan Zimmerman=22 &lt;<a href=3D=22mail=
to:jordan=40jordanzimmerman.com=22>jordan=40jordanzimmerman.com</a>&gt;
wrote:<br>
<br></span></div>
<blockquote type=3D=22cite=22>
<div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family:Helvetica,Arial=
;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;=22=
>
<span>Which version of enter() are you using=3F I see a potential bug
when the no arg version of enter() is used.</span></div>
<div id=3D=22bloop=5Fsign=5F1395786119598438144=22 class=3D=22bloop=5Fsig=
n=22>
<div style=3D=22font-family:helvetica,arial;font-size:13px=22>
<span><br></span></div>
</div>
<div style=3D=22color:black=22><span><br>
=46rom:&nbsp;<span style=3D=22color:black=22>Brian Phillips</span> <a hre=
f=3D=22mailto:brian=40etinternational.com=22>brian=40etinternational.com<=
/a><br>

Reply:&nbsp;<span style=3D=22color:black=22>Brian Phillips</span>
<a href=3D=22mailto:brian=40etinternational.com=22>brian=40etinternationa=
l.com</a><br>

Date:&nbsp;<span style=3D=22color:black=22>March 25, 2014 at 4:19:36
PM</span><br>
To:&nbsp;<span style=3D=22color:black=22>Jordan Zimmerman</span> <a href=3D=
=22mailto:jordan=40jordanzimmerman.com=22>jordan=40jordanzimmerman.com</a=
><br>

Subject:&nbsp; <span style=3D=22color:black=22>Re: Curator barriers
missing watch events<br></span></span></div>
<br>
<blockquote type=3D=22cite=22 class=3D=22clean=5Fbq=22>
<div style=3D=22word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-=
line-break: after-white-space;=22>
<div>
<div><span>Good idea, but yes I am. The connection state doesn=E2=80=99t
change while I=E2=80=99m executing the barrier code. It seems to be some
kind of race condition I think, as sometimes it work and sometimes
it doesn=E2=80=99t. I=E2=80=99ve looked through the recipe code and it lo=
oks good
as far as I can tell though. I=E2=80=99m practically pulling my hair out =
at
this point.</span></div>
<div><span><br></span></div>
<div><span>I may try a non-curator zookeeper only barrier tomorrow.
See if that works. Or I may start trying to debug the zookeeper
client, see if its actually getting the watches but not delivering
them.</span></div>
<div><span><br></span></div>
<div><span>=5FB</span></div>
<span><br></span>
<div>
<div><span>On Mar 25, 2014, at 4:54 PM, Jordan Zimmerman
&lt;<a href=3D=22mailto:jordan=40jordanzimmerman.com=22>jordan=40jordanzi=
mmerman.com</a>&gt;
wrote:</span></div>
<span><br class=3D=22Apple-interchange-newline=22></span>
<blockquote type=3D=22cite=22>
<div style=3D=22font-family: Helvetica, Arial; font-size: 13px; font-styl=
e: normal; font-variant: normal; font-weight: normal; letter-spacing: nor=
mal; line-height: normal; orphans: auto; text-align: start; text-indent: =
0px; text-transform: none; white-space: normal; widows: auto; word-spacin=
g: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nb=
sp-mode: space; -webkit-line-break: after-white-space;=22>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family: Helvetica, Ari=
al; font-size: 13px; margin: 0px;=22>
<span>Are you setting a ConnectionStateListener=3F If the connection
gets SUSPENDED or LOST then you=E2=80=99d need to reinitialize your
barrier.</span></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family: Helvetica, Ari=
al; font-size: 13px; margin: 0px;=22>
<span><br></span></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family: Helvetica, Ari=
al; font-size: 13px; margin: 0px;=22>
<span>-JZ</span></div>
<div id=3D=22bloop=5Fsign=5F1395780854857120000=22 class=3D=22bloop=5Fsig=
n=22>
<div style=3D=22font-family: helvetica, arial; font-size: 13px;=22>
<span><br></span></div>
</div>
<div style=3D=22=22><span><br>
=46rom:&nbsp;<span style=3D=22=22>Brian Phillips</span><span class=3D=22A=
pple-converted-space=22>&nbsp;</span><a href=3D=22mailto:brian=40etintern=
ational.com=22>brian=40etinternational.com</a><br>

Reply:&nbsp;<span style=3D=22=22><a href=3D=22mailto:user=40curator.apach=
e.org=22>user=40curator.apache.org</a></span><span class=3D=22Apple-conve=
rted-space=22>&nbsp;</span><a href=3D=22mailto:user=40curator.apache.org=22=
>user=40curator.apache.org</a><br>

Date:&nbsp;<span style=3D=22=22>March 25, 2014 at 2:51:42 PM</span><br>
To:&nbsp;<span style=3D=22=22><a href=3D=22mailto:user=40curator.apache.o=
rg=22>user=40curator.apache.org</a></span><span class=3D=22Apple-converte=
d-space=22>&nbsp;</span><a href=3D=22mailto:user=40curator.apache.org=22>=
user=40curator.apache.org</a><br>

Subject:&nbsp;<span style=3D=22=22><span class=3D=22Apple-converted-space=
=22>&nbsp;</span>Re: Curator barriers missing
watch events<span class=3D=22Apple-converted-space=22>&nbsp;</span><br></=
span></span></div>
<br>
<blockquote type=3D=22cite=22 class=3D=22clean=5Fbq=22>
<div style=3D=22word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-=
line-break: after-white-space;=22>
<div>
<div><span>I have tried writing a test program which launches two
programs in the same manor, each makes a connection then loops over
barriers with a Thread.sleep(random) in-between. This run
indefinitely and everything works out fine.</span></div>
<div><span><br></span></div>
<div><span>I have also tried writing my own barrier, which uses a
SharedCount, where each guy tries to increment it until it hits a
memberQty. This too missed watch events and does not work
properly.</span></div>
<div><span><br></span></div>
<div><span>It=E2=80=99s almost as if something else that I=E2=80=99ve don=
e during
the running of my program has broken zookeepers watch events
somehow. Is there any good way to debug watch events in general=3F
I=E2=80=99ve tried to look at the DEBUG output for my zookeeper server lo=
g,
but it looks the same for the working vs non-working
barriers...</span></div>
<div><span><br></span></div>
<div><span>=5FB</span></div>
<span><br></span>
<div>
<div><span>On Mar 25, 2014, at 3:42 PM, Jordan Zimmerman
&lt;<a href=3D=22mailto:jordan=40jordanzimmerman.com=22>jordan=40jordanzi=
mmerman.com</a>&gt;
wrote:</span></div>
<span><br class=3D=22Apple-interchange-newline=22></span>
<blockquote type=3D=22cite=22>
<div style=3D=22font-family: Helvetica, Arial; font-size: 13px; font-styl=
e: normal; font-variant: normal; font-weight: normal; letter-spacing: nor=
mal; line-height: normal; orphans: auto; text-align: start; text-indent: =
0px; text-transform: none; white-space: normal; widows: auto; word-spacin=
g: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nb=
sp-mode: space; -webkit-line-break: after-white-space;=22>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family: Helvetica, Ari=
al; font-size: 13px; margin: 0px;=22>
<span>Unfortunately, the barrier recipes aren=E2=80=99t widely used (from=

what I know). So, there may well be a bug. If you could get a test
to show the problem that would be ideal.</span></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family: Helvetica, Ari=
al; font-size: 13px; margin: 0px;=22>
<span><br></span></div>
<div id=3D=22bloop=5Fcustomfont=22 style=3D=22font-family: Helvetica, Ari=
al; font-size: 13px; margin: 0px;=22>
<span>-JZ</span></div>
<div id=3D=22bloop=5Fsign=5F1395776518916934912=22 class=3D=22bloop=5Fsig=
n=22>
<div style=3D=22font-family: helvetica, arial; font-size: 13px;=22>
<span><br></span></div>
</div>
<div><span><br>
=46rom:&nbsp;<span>Brian Phillips</span><span class=3D=22Apple-converted-=
space=22>&nbsp;</span><a href=3D=22mailto:brian=40etinternational.com=22>=
brian=40etinternational.com</a><br>

Reply:&nbsp;<span><a href=3D=22mailto:user=40curator.apache.org=22>user=40=
curator.apache.org</a></span><span class=3D=22Apple-converted-space=22>&n=
bsp;</span><a href=3D=22mailto:user=40curator.apache.org=22>user=40curato=
r.apache.org</a><br>

Date:&nbsp;<span>March 25, 2014 at 2:38:40 PM</span><br>
To:&nbsp;<span><a href=3D=22mailto:user=40curator.apache.org=22>user=40cu=
rator.apache.org</a></span><span class=3D=22Apple-converted-space=22>&nbs=
p;</span><a href=3D=22mailto:user=40curator.apache.org=22>user=40curator.=
apache.org</a><br>

Subject:&nbsp;<span><span class=3D=22Apple-converted-space=22>&nbsp;</spa=
n>Curator barriers missing watch
events<span class=3D=22Apple-converted-space=22>&nbsp;</span><br></span><=
/span></div>
<br>
<blockquote type=3D=22cite=22 class=3D=22clean=5Fbq=22><span>Hi guys,&nbs=
p;<br>
<br>
I=E2=80=99ve been integrating curator into my project and have recently r=
un
into an issue I just can=E2=80=99t seem to make sense of.&nbsp;<br>
<br>
I=E2=80=99m running two JVMs on the same host machine, each with their ow=
n
curator connection. At the beginning of my program I=E2=80=99m using the
DistributedDoubleBarrier recipe, and once again at the end of my
program. A bunch of work is done in-between, including zookeeper
set/get/watches of other nodes.&nbsp;<br>
<br>
I=E2=80=99m finding that the first double barrier, everyone always making=

it through. The job-end barrier, sometimes everyone gets through,
but more often than not one of the programs hangs in enter's
wait(), and never gets the watch event for the ready path which
notifies it to proceed. If I look in zookeeper, I can see that the
ready path is actually set in there.&nbsp;<br>
<br>
It would seem that the watch for one of the programs just never
triggers.&nbsp;<br>
<br>
To simplify debugging, I=E2=80=99ve set both double barriers to only ever=

call enter() and not leave(). Both barriers have their own separate
path.&nbsp;<br>
<br>
Also, the program never shuts down or disconnects from zookeeper.
It just sleeps infinitely after it gets out of the final
barrier.&nbsp;<br>
<br>
Any idea on how to debug this issue=3F I don=E2=80=99t mind hacking up
zookeeper/curator code to insert my own debugging statements if it
comes to that.&nbsp;<br>
<br>
=5FBrian=3D</span></blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br></div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>


</div></div></span></blockquote></body></html>
--53346042_15b5af5c_14f7--