kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neha Narkhede (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-233) The producer's load balancing logic can send requests to dead brokers, when using the async producer option
Date Tue, 03 Jan 2012 19:20:40 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178954#comment-13178954
] 

Neha Narkhede commented on KAFKA-233:
-------------------------------------

Al though this JIRA is marked with the newbie tag, this is amongst the harder of all the newbie
JIRAs. Anyone who is interested in knowing the Producer logic inside out, can give this a
try. It will be a very good enhancement to the producer logic
                
> The producer's load balancing logic can send requests to dead brokers, when using the
async producer option
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-233
>                 URL: https://issues.apache.org/jira/browse/KAFKA-233
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Neha Narkhede
>              Labels: newbie
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> The ZK producer, when used with the async producer option does the following 
> 1. Create a pool of async producers, one each for a broker registered under /broker/ids
> 2. On each send request, apply the Partitioner, to decide the broker and partition to
send the data
> 3. Use the Async producer's send API to enqueue that data into the async producer's queue
> 4. When the data is dequeued by the ProducerSendThread, use the underlying sync producer
to send it to the broker
> The load balancing decision is taken in step 2, before entering the queue. This leaves
a window of error, equal to the queue length, when a broker can go down. When this happens,
potentially, a queue worth of data can fail to reach a broker, and will be dropped by the
EventHandler. 
> To correct this, the Producer, with the async option, needs to be refactored to allow
only a single queue to hold all requests. And the application of the Partitioner should be
moved to the end of the queue, in the EventHandler.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message