spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-2383) With auto.offset.reset, KafkaReceiver potentially deletes Consumer nodes from Zookeeper
Date Tue, 25 Nov 2014 14:14:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224571#comment-14224571
] 

Tathagata Das commented on SPARK-2383:
--------------------------------------

This issue has been resolved in Spark 1.2.0. I am marking this JIRA as duplicate of that SPARK-2492
and closing this.

> With auto.offset.reset, KafkaReceiver potentially deletes Consumer nodes from Zookeeper
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-2383
>                 URL: https://issues.apache.org/jira/browse/SPARK-2383
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>            Reporter: Tobias Pfeiffer
>
> When auto.offset.reset is set in the Kafka configuration, then {{KafkaReceiver}}'s {{tryZookeeperConsumerGroupCleanup()}}
will delete the whole /consume/<groupId> tree in Zookeeper before creating consumer
nodes. If there are already consumer nodes present (this may happen when multiple KafkaReceivers
in the same consumer group are launched), they are deleted as well, leading to subsequent
NoNode exceptions, for example, on rebalance.
> There should be a check before the delete like {{if (zk.countChildren(dir + "/ids") ==
0) ...}} (ideally in an atomic way) in order to prevent deleting existing consumer nodes.
> (Also note that the behavior of auto.offset.reset as realized by Spark's Kafka receiver
differs from the behavior defined in Kafka's documentation.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message