kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikumar (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (KAFKA-2565) Offset Commit is not working if multiple consumers try to commit the offset
Date Thu, 24 Aug 2017 15:52:01 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Manikumar resolved KAFKA-2565.
------------------------------
    Resolution: Cannot Reproduce

may be related to deployment issue. Pl reopen if you think the issue still exists


> Offset Commit is not working if multiple consumers try to commit the offset
> ---------------------------------------------------------------------------
>
>                 Key: KAFKA-2565
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2565
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.8.1, 0.8.2.1, 0.8.2.2
>            Reporter: Sreenivasulu Nallapati
>            Assignee: Neha Narkhede
>
> We are seeing some strange behaviour with commitOffsets() method of kafka.javaapi.consumer.ConsumerConnector.
We committing the offsets to zookeeper at the end of the consumer batch. We are running multiple
consumers for the same topic.
> Test details: 
> 1.	Created a topic with three partitions
> 2.	Started three consumers (cronjob) at the same time. The aim is that each consumer
to process one partition.
> 3.	Each consumer at the end of the batch, it will call the commitOffsets() method on
kafka.javaapi.consumer.ConsumerConnector
> 4.	The offsets are getting properly updated in zookeeper if we run the consumers for
small set (say 1000 messages) of messages.
> 5.	But for larger number of messages, commit offset is not working as expected…sometimes
only two offsets are properly committing and other one remains as it was.
> 6.	Please see the below example
> Partition: 0 Latest Offset: 1057585
> Partition: 1 Latest Offset: 1057715
> Partition: 2 Latest Offset: 1057590
> Earliest Offset after all consumers completed: {0=1057585, 1=724375, 2=1057590}
> Highlighted in red supposed to be committed as 1057715 but it did not.
> Please check if it is bug with multiple consumers. When multiple consumers are trying
to update the same path in Zookeper, is there any synchronization issue?
> Kafka Cluster details
> 1 zookeeper
> 3 brokers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message