hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: ephemeral node after server bounce
Date Thu, 04 Feb 2010 22:47:29 GMT

Yonik Seeley wrote:
> We have solr nodes create ephemeral znodes (name based on host and port).
> The ephemeral znode takes some time to remove of course, so what
> happens is that if I bounce a solr server (containing a zk client) the
> ephemeral node will still exist when the server comes back up.  Since
> it exists, the ephemeral won't be re-created, but it does disappear
> later.

Yes. This is expected behavior - and one of the main reasons for having 
low session timeouts. Unfortunately the alternative of having lower 
timeout can also have negative effect (sensitivity to network/resource 
issues, in particular GC as you know)

> What's the best way to handle this situation?  Delete and re-create?
> Watch it and re-create when it does disappear?
> There's no way to "hand over" responsibility for an ephemeral znode, right?

There is one way, not really "hand over" but rather "take back".

What I mean is, when you establish a session you get a session id AND a 
password. The session is valid until either it is closed or expires. If, 
when your client (solr server) comes back up, it is able to provide the 
session id and password to the ZK service it will _recover_ the session. 
This probably means that you would have to write the sessionid/password 
to disk (if your client is crashing/killed for example) and handle some 
corner cases (like session expired). It's not a common case as you have 
to store the id persistently, but it is possible and might address your 
issue (assuming the primary issue is that your client goes down and 
becomes active again quickly).

A second alternative is to actively delete the znode in question. 
However this has similar issues to the prior paragraph - the client 
that's coming up would have to know the session id is had previously, so 
that it could get a Stat on the znode, compare the "ephemeralOwner" and 
delete if it was the previous owner.

However short of knowing the session id I don't see how you get around 
this issue other than just waiting for the timeout. The problem is how 
would you know to delete the ephemeral node in question? How is any 
other solr client going to know that the client died, and therefore 
remove the znode... Really that's what you are relying on zk for, via 
the timeout.

Yes, you could set a watch on the znode in question, when it is deleted 
(via the eventual session timeout) you would recreate.

Another option is that the client that's being restarted, it could write 
it's IP address somewhere (as part of the znode or part of the data say, 
or another znode that names ref to this znode, etc...). If the client 
was bounced it could compare it's IP to the ip "assigned" to the znode, 
if it was the same you would have confidence you were the owner (this 
one depends on the usecase but based on what you described it should 
work unless the client changes IPs dynamically).

Patrick





Mime
View raw message