kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matan Safriel (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-789) Producer-side persistence for delivery guarantee
Date Tue, 05 Mar 2013 19:32:12 GMT
Matan Safriel created KAFKA-789:

             Summary: Producer-side persistence for delivery guarantee
                 Key: KAFKA-789
                 URL: https://issues.apache.org/jira/browse/KAFKA-789
             Project: Kafka
          Issue Type: Improvement
          Components: producer 
            Reporter: Matan Safriel
            Assignee: Jun Rao
            Priority: Minor
             Fix For: 0.9

A suggestion for higher guarantee for the part of entering messages into Kafka through it's
producer. It aims to address the case that the entire set of broker replicas for a topic and
partition is not available. Currently, in that case, data is lost. When a message set exhausts
the send retry counter, the message set will be simply dropped. It would be nice being able
to provide higher guarantee that a message passed to the producer would eventually be received
by the broker. 

In an environment with some disk space to spare for this on the producer side, persisting
to disk would seem to enable keeping messages for later retry (until defined space limits
are exhausted). Thus somewhat elevating the level of guarantee. 

One way to facilitate this would be capitalizing on https://issues.apache.org/jira/browse/KAFKA-496,
as the feedback it will add will enable knowing what needs to be retried again later. Changes
to the producer or a wrapper around it (that may require access to the partitioning functions)
would be able to persist failed message sets and manage delivery with a nice level of guarantee.
As it would affect performance and use disks, should probably be a non-default option.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message