Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of fpjunqueira@yahoo.com
 designates 98.139.212.187 as permitted sender)
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_E08DD13B-A104-4F39-B515-E32DDBC2E59D"
Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\))
Subject: Re: cluster/ephemeral nodes inconsistency
From: Flavio Junqueira <fpjunqueira@yahoo.com.INVALID>
In-Reply-To: 
 <77166901.1003416.1421239240841.JavaMail.yahoo@jws10630.mail.bf1.yahoo.com>
Date: Wed, 14 Jan 2015 21:30:15 +0000
Cc: "user@zookeeper.apache.org" <user@zookeeper.apache.org>
Message-Id: <9B6081EB-8AC6-45E3-9036-063E9210B6AF@yahoo.com>
References: 
 <CAHtp0sM4mP-tHYHB5MQmTx4JK=d9AZXpPmVi6d5kkfHtn4nGgQ@mail.gmail.com>
 <77166901.1003416.1421239240841.JavaMail.yahoo@jws10630.mail.bf1.yahoo.com>
To: Flavio Junqueira <fpjunqueira@yahoo.com>

--Apple-Mail=_E08DD13B-A104-4F39-B515-E32DDBC2E59D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Also, what was the last operation that changed the messed up znode and =
when has the operation been executed?

-Flavio

> On 14 Jan 2015, at 12:40, Flavio Junqueira <fpjunqueira@yahoo.com> =
wrote:
>=20
> But you do observe the session being closed, yes? And the ephemeral =
can be listed with getChildren but you can't get it with getData, is it =
right?
>=20
> -Flavio
>=20
>=20
> On Wednesday, January 14, 2015 11:42 AM, Kuba Lekstan =
<kuebzky@gmail.com> wrote:
>=20
>=20
> German, today it had happen on our secondary cluster which consist of =
3
> nodes, the leader didn't see the node but two other followers did.
>=20
> Flavio, I browsed the logs but was unable to find anything =
interesting,
> only setData operations were issued.
>=20
> Problematic znode was last modified at 13 Jan 2015 17:xx, we have =
noticed
> the issue at 14 Jan 2015 11:xx.
>=20
> 2015-01-14 10:52 GMT+01:00 Flavio Junqueira =
<fpjunqueira@yahoo.com.invalid <mailto:fpjunqueira@yahoo.com.invalid>>:
>=20
> > Hi there,
> > I suggest a couple of things here:
> > - Use LogFormatter to look into the transaction logs to check the
> > operations that are actually coming across.- It would be nice be =
able to
> > reproduce it outside your app, ideally as a junit test so that we =
can start
> > working on it.
> > I vaguely remember coming across such a problem, but I'll need to =
dig into
> > it. Does anyone on this list recall a similar problem?
> > -Flavio
> >
> >      On Wednesday, January 14, 2015 9:14 AM, Kuba Lekstan <
> > kuebzky@gmail.com <mailto:kuebzky@gmail.com>> wrote:
> >
> >
> >
> >  German do you have any idea what might be causing these? Today same =
issue
> > had happen.
> >
> > 2014-11-21 5:42 GMT+01:00 Yogesh Patil <patyogesh@gmail.com =
<mailto:patyogesh@gmail.com>>:
> >
> > > Hi Zookeepers,
> > > I am also experiencing the similar problem since yestderday. I =
have
> > pretty
> > > much similar setup and ephemeral znodes in place for keep-alive =
kind of
> > > function. I too see in spite of ZK session going down, ephemeral =
znodes
> > > still LIVES.
> > >
> > > I am using ZK 3.5.0.
> > >
> > > Any solution/fix for this type of an issue??
> > >
> > >
> > > --
> > > Sincerely,
> > >
> > > *Yogesh Patil*
> > >
> > >
> > >
> > > On Thu, Nov 13, 2014 at 2:10 PM, Kuba Lekstan <kuebzky@gmail.com =
<mailto:kuebzky@gmail.com>> wrote:
> > >
> > > > Sorry, forgot to mention. Version: 3.4.6.
> > > >
> > > > Thanks.
> > > >
> > > > 2014-11-13 18:11 GMT+01:00 German Blanco <
> > german.blanco.blanco@gmail.com =
<mailto:german.blanco.blanco@gmail.com>
> > > >:
> > > >
> > > > > Hello,
> > > > >
> > > > > which version of Zookeeper are you using?
> > > > >
> > > > > On Thu, Nov 13, 2014 at 5:25 PM, Kuba Lekstan =
<kuebzky@gmail.com <mailto:kuebzky@gmail.com>>
> > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > A bit of details:
> > > > > > We have 5 node cluster, which we use for configuration =
distrubution
> > > and
> > > > > > monitoring active instances of our applications. Each =
application
> > > > creates
> > > > > > its ephemeral node, so we know which apps are alive, how =
many of
> > them
> > > > > there
> > > > > > is and what they are doing.
> > > > > >
> > > > > > The problem had happen at 4th November, first time it was =
around
> > 4AM,
> > > > > > second time around 12PM.
> > > > > > First time it was middle of the night when I got woken up, =
the
> > > support
> > > > > guys
> > > > > > told me that something is wrong with config distribution.
> > > > > >
> > > > > > First I've checked apps for errors but didn't find anything
> > > > interesting,
> > > > > > then I looked at what's in zookeeper (using =
node-zk-browser).
> > > > > > I've noticed that there are 3 ephemeral nodes which were =
created at
> > > 1st
> > > > > nov
> > > > > > (while the oldest application was started on 3rd nov), I =
could read
> > > its
> > > > > > data but was not able to delete them - was getting NONODE
> > exception.
> > > > > >
> > > > > > I thought wtf - why I cannot delete these nodes, something =
very bad
> > > had
> > > > > to
> > > > > > happen with ZK.
> > > > > >
> > > > > > So I sshed on the leader and using CLI I tried to read these =
nodes
> > > but
> > > > I
> > > > > > was not able to - the leader was telling me that such nodes =
doesn't
> > > > > exist.
> > > > > > After this I started to ssh to the rest of the nodes in =
cluster and
> > > > > trying
> > > > > > to read these nodes. Finally I found the server which did =
let me
> > read
> > > > the
> > > > > > data of these nodes.
> > > > > > Because of the inconsistency I've decided to restart it. =
Restart
> > did
> > > > > help,
> > > > > > everything went back to normal state. The ephemeral nodes
> > > disappeared.
> > > > > >
> > > > > > Similar situation had happen at 12PM but this time I had a =
lot more
> > > > time
> > > > > to
> > > > > > look what is wrong. Second time the problem was about 3 =
ephemeral
> > > nodes
> > > > > > which were created at 1st now (again?). This time I dig a =
bit
> > deeper
> > > > and
> > > > > > look into logs and 4 letter commands - but could not find =
anything
> > > > > > interesting except the all these 3 nodes were created under
> > different
> > > > > > sessionids but zk had no hosts connected under this =
sessionids.
> > > > > > Solution was similar to the one from 4AM but this time I've =
delete
> > > all
> > > > > > files in ZK data directory.
> > > > > >
> > > > > > Oddly enough the problem happened twice on the same ZK node, =
the
> > > final
> > > > > > solution was to clear ZK data directory. After clearing the
> > directory
> > > > the
> > > > > > problem didn't happen again.
> > > > > >
> > > > > > I tried to look for solution/similar problems, I found the =
posts
> > > where
> > > > > > people were complaining about ephemeral nodes not being =
removed
> > after
> > > > > > client session gets closed. But I was not able to find posts =
about
> > ZK
> > > > not
> > > > > > being consistent.
> > > > > >
> > > > > > What do you think about this? Can we do something to fix =
this?
> > > > > >
> > > > > > Sorry for my english, I was doing my best. :)
> > > > > >
> > > > > > Thanks, Kuba.
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
> >
>=20
>=20


--Apple-Mail=_E08DD13B-A104-4F39-B515-E32DDBC2E59D--