kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Producer-side persistence and re-sending
Date Mon, 04 Mar 2013 16:59:40 GMT

Thanks for picking this up. Yes, this is likely a post 0.8 item and should
go into trunk. Could you file a jira and post your initial design there for


On Sat, Mar 2, 2013 at 12:53 PM, matan <matan@cloudaloe.org> wrote:

> Hi,
> I am designing a patch on top the 0.8 code base.
> The patch would provide persistence on the producer side. Meaning that
> messages being passed to the producer are persisted rather than kept
> transiently in memory. So that if the broker/s cannot be reached, messages
> can  accumulate and will be sent through to the broker/s, when they are
> available again. Although this would be somewhat superfluous in the new
> replication paradigm of 0.8, it's still possible to have some failures that
> disconnect a producer from the entire set of brokers. In that case, this
> patch-under-design would prevent data loss. Making the pipeline even more
> secured, and relieving producers of the need to handle persistence on their
> own. Plan is to use the Kafka Logger component for that. Of course having
> this behavior completely optional through a configuration option.
> The slightly-deeper level design details are to use a Kafka Log per topic
> & partition (otherwise given the existing 0.8 code, in and around
> /producer.async.**DefaultEventHandler.**dispatchSerializedData/, it would
> seem resource intensive to keep track of messages sent v.s. failed ones for
> managing resending). Using the logger, behavior for keeping replica sets in
> sync would be skipped through choice of parameters, or would be made
> parameterized to fully neutralize it regarding the producer's own logging.
> Now of course, given that 0.8 seems to be far along its runway, I assume
> this should go on top the trunk, which I'd like to confirm with you is
> where post 0.8 lives.
> I'd appreciate your comments...
> Thanks,
> Matan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message