incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neha Narkhede (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-233) The producer's load balancing logic can send requests to dead brokers, when using the async producer option
Date Tue, 03 Jan 2012 19:08:41 GMT
The producer's load balancing logic can send requests to dead brokers, when using the async
producer option
-----------------------------------------------------------------------------------------------------------

                 Key: KAFKA-233
                 URL: https://issues.apache.org/jira/browse/KAFKA-233
             Project: Kafka
          Issue Type: Bug
          Components: core
    Affects Versions: 0.7
            Reporter: Neha Narkhede


The ZK producer, when used with the async producer option does the following 

1. Create a pool of async producers, one each for a broker registered under /broker/ids
2. On each send request, apply the Partitioner, to decide the broker and partition to send
the data
3. Use the Async producer's send API to enqueue that data into the async producer's queue
4. When the data is dequeued by the ProducerSendThread, use the underlying sync producer to
send it to the broker

The load balancing decision is taken in step 2, before entering the queue. This leaves a window
of error, equal to the queue length, when a broker can go down. When this happens, potentially,
a queue worth of data can fail to reach a broker, and will be dropped by the EventHandler.


To correct this, the Producer, with the async option, needs to be refactored to allow only
a single queue to hold all requests. And the application of the Partitioner should be moved
to the end of the queue, in the EventHandler.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message