kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jun Rao (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-228) Reduce duplicate messages served by the kafka consumer for uncompressed topics to 0
Date Wed, 04 Jan 2012 00:06:40 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jun Rao updated KAFKA-228:
--------------------------

    Attachment: kafka-228_v2.patch

Attach patch v2. It implements ConsumerIterator in an alternative way that doesn't require
locking.

To apply, apply patch v1, revert ConsumerIterator, then apply patch v2.
                
> Reduce duplicate messages served by the kafka consumer for uncompressed topics to 0
> -----------------------------------------------------------------------------------
>
>                 Key: KAFKA-228
>                 URL: https://issues.apache.org/jira/browse/KAFKA-228
>             Project: Kafka
>          Issue Type: Improvement
>    Affects Versions: 0.7
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>             Fix For: 0.7.1
>
>         Attachments: kafka-228.patch, kafka-228_v2.patch
>
>
> Kafka guarantees at-least once delivery of messages.The high level consumer provides
highly available partitioned consumption of data within the same consumer group. In the event
of broker failures or consumer failures within a group, the high level consumer rebalances
and redistributes the topic partitions evenly amongst the consumers in a group. With the current
design, during this rebalancing operation, Kafka introduces duplicates in the consumed data.

> This JIRA improves the rebalancing operation and the consumer iterator design to guarantee
0 duplicates while consuming uncompressed topics. There will be a small number of duplicates
while serving compressed data, but it will be bound by the compression batch size.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message