kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christine Wright (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4277) creating ephemeral node already exist
Date Wed, 12 Jul 2017 17:50:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084382#comment-16084382
] 

Christine Wright commented on KAFKA-4277:
-----------------------------------------

I have also seen this issue,

zookeeper 3.4.6  cluster of 3
kafka_2.11-0.10.1.1 cluster of 5

I have full zook and kafka logs, but I'm not sure they will provide more information that
what was provided above. 

There was a period of network instability with lots of zookeeper timeouts and reconnects on
the entire kafka cluster, but all was fine until one broker hit this problem

ERROR Error handling event ZkEvent[New session event sent to kafka.server.KafkaHealthcheck$SessionExpireListener@2a31b49a]
(org.I0Itec.zkclient.ZkEventThread)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/4. This
probably indicates that you either have configured a brokerid that is already in use, or else
you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears
to be re-registering.
	at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:393)
	

and on zookeeper I see:
2017-07-08 00:02:24,478 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@645]
- Got user-level KeeperException when processing sessionid:0x5d205cdd1f0000 type:create cxid:0x6
zxid:0x900000012 txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NodeExists
for /brokers
2017-07-08 00:02:24,478 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@645]
- Got user-level KeeperException when processing sessionid:0x5d205cdd1f0000 type:create cxid:0x7
zxid:0x900000013 txntype:-1 reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode = NodeExists
for /brokers/ids
2017-07-08 00:02:24,478 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@645]
- Got user-level KeeperException when processing sessionid:0x5d205cdd1f0000 type:create cxid:0x8
zxid:0x900000014 txntype:-1 reqpath:n/a Error Path:/brokers/ids/4 Error:KeeperErrorCode =
NodeExists for /brokers/ids/4


> creating ephemeral node already exist
> -------------------------------------
>
>                 Key: KAFKA-4277
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4277
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.10.0.0
>            Reporter: Feixiang Yan
>
> I use zookeeper 3.4.6.
> Zookeeper session time out, zkClient try reconnect failed. Then re-establish the session
and re-registering broker info in ZK, throws NODEEXISTS Exception.
>  I think it is because the ephemeral node which created by old session has not removed.

> I read the [ZkUtils.scala|https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/utils/ZkUtils.scala]
of 0.8.1, createEphemeralPathExpectConflictHandleZKBug try create node in a while loop until
create success. This can solve the issue. But in [ZkUtils.scala|https://github.com/apache/kafka/blob/0.10.0.1/core/src/main/scala/kafka/utils/ZkUtils.scala]
 0.10.1 the function removed.
> {noformat}
> [2016-10-07 19:00:32,562] INFO Socket connection established to 10.191.155.238/10.191.155.238:21819,
initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,563] INFO zookeeper state changed (Expired) (org.I0Itec.zkclient.ZkClient)
> [2016-10-07 19:00:32,564] INFO Unable to reconnect to ZooKeeper service, session 0x1576b11f9b201bd
has expired, closing socket connection (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,564] INFO Initiating client connection, connectString=10.191.155.237:21819,10.191.155.238:21819,10.191.155.239:21819/cluster2
sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@ae71be2 (org.apache.zookeeper.ZooKeeper)
> [2016-10-07 19:00:32,566] INFO Opening socket connection to server 10.191.155.237/10.191.155.237:21819.
Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,566] INFO Socket connection established to 10.191.155.237/10.191.155.237:21819,
initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,566] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,567] INFO Session establishment complete on server 10.191.155.237/10.191.155.237:21819,
sessionid = 0x1579ecd39c20006, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,567] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
> [2016-10-07 19:00:32,608] INFO re-registering broker info in ZK for broker 3 (kafka.server.KafkaHealthcheck$SessionExpireListener)
> [2016-10-07 19:00:32,610] INFO Creating /brokers/ids/3 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
> [2016-10-07 19:00:32,611] INFO Result of znode creation is: NODEEXISTS (kafka.utils.ZKCheckedEphemeral)
> [2016-10-07 19:00:32,614] ERROR Error handling event ZkEvent[New session event sent to
kafka.server.KafkaHealthcheck$SessionExpireListener@324f1bc] (org.I0Itec.zkclient.ZkEventThread)
> java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/3.
This probably indicates that you either have configured a brokerid that is already in use,
or else you have shutdown this broker and restarted it faster than the zookeeper timeout so
it appears to be re-registering.
>         at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:305)
>         at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:291)
>         at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)
>         at kafka.server.KafkaHealthcheck$SessionExpireListener.handleNewSession(KafkaHealthcheck.scala:104)
>         at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:735)
>         at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message