Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (athena.apache.org: domain of jlent@tivo.com designates
 208.73.180.201 as permitted sender)
Received-SPF: Pass (sjimx1.tivo.com: domain of jlent@tivo.com
  designates 208.73.180.16 as permitted sender)
  identity=mailfrom; client-ip=208.73.180.16;
  receiver=sjimx1.tivo.com; envelope-from="jlent@tivo.com";
  x-sender="jlent@tivo.com"; x-conformance=spf_only;
  x-record-type="v=spf1"
Received-SPF: None (sjimx1.tivo.com: no sender authenticity
  information available from domain of
  postmaster@sjmailman01.tivo.com) identity=helo;
  client-ip=208.73.180.16; receiver=sjimx1.tivo.com;
  envelope-from="jlent@tivo.com";
  x-sender="postmaster@sjmailman01.tivo.com";
  x-conformance=spf_only
Message-ID: <542D7E59.1090004@digitalsmiths.com>
Date: Thu, 2 Oct 2014 12:33:29 -0400
From: James Lent <jlent@digitalsmiths.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.1.1
MIME-Version: 1.0
To: <user@zookeeper.apache.org>
Subject: Question Regarding Ephemeral Nodes And New Sessions (Related To
 KAFKA-1387)
Content-Type: multipart/alternative;
	boundary="------------000205010802000005060907"

--------------000205010802000005060907
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: quoted-printable

I have run into the issue documented by KAFKA-1387 and have been trying to =
come up with a solution.  A summary of this issue is:

  *   Kafka registers brokers and consumers via ephemeral ZooKeeper nodes.
  *   When a connection fails and an Expire event is received Kafka reconne=
cts and then attempts to recreate these ephemeral nodes.
  *   If the node still/already exists when the Kafka attempts to recreate =
it Kafka currently assumes (I am working with 0.8.1.1) that ZooKeeper is ju=
st slow deleting the node from the old session and therefore goes into a de=
lay loop waiting for ZooKeeper to remove the stale node so it can create a =
new ephemeral associated with the new session.
  *   In my stress testing I have seen cases where the connection can fail =
multiple times in a short period of time and if one of these failures occur=
s while handling the Expire event Kafka can end up with a backlog of two or=
 more Expire events.  When the first of these finally gets processed it rec=
reates the node against the latest session.  However, when the next one is =
processed the Kafka broker or consumer goes into a never ending delay loop =
waiting for the stable node to go away.  This will not happen unless the co=
nnection fails again, but, then the process just repeats itself.

I proposed a fix in the KAFKA-1387 Jira issue to generate some discussion o=
f potential fixes for this issue. One of the Kafka developers requested tha=
t I vet the basic assumption of my fix with the ZooKeeper team.  My solutio=
n is basically:

  *   Register (via ZkClient) for notifications of both session and node ev=
ents
  *   When processing the Expire event:
     *   If the node does not exist then recreate the node (current behavio=
r)
     *   If the node exists do nothing (no looping)
  *   When processing a delete node event:
     *   If the node does not exist then recreate the node (new behavior)
     *   If the node exists do nothing

The basic assumption is that:

    "In the rare case where the node still exists from the previous session=
 when the Expire message is processed then we can be confident that we will=
 be notified later when the node is finally deleted."

In my testing I have seen:

  *   If I recreate the node while handling the Expire I do not later get a=
 delete message (for the already deleted node).
  *   If I do nothing when I process the Expire (to partially simulate a sl=
ow ZooKeeper delete) then I do get a delete message for the old node (which=
 was actually deleted before I processed the Expire message).

I would greatly appreciate your insights on this issue.  For more details y=
ou can see the Kafka issue.

James Lent
Senior Software Engineer

Digitalsmiths
A TiVo Company

www.digitalsmiths.com<http://www.digitalsmiths.com/>
jlent@digitalsmiths.com<mailto:jlent@digitalsmiths.com>  | office 919.460.4=
747

________________________________

This email and any attachments may contain confidential and privileged mate=
rial for the sole use of the intended recipient. Any review, copying, or di=
stribution of this email (or any attachments) by others is prohibited. If y=
ou are not the intended recipient, please contact the sender immediately an=
d permanently delete this email and any attachments. No employee or agent o=
f TiVo Inc. is authorized to conclude any binding agreement on behalf of Ti=
Vo Inc. by email. Binding agreements with TiVo Inc. may only be made by a s=
igned written agreement.

--------------000205010802000005060907--