kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Kreps (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-736) Add an option to the 0.8 producer to mimic 0.7 producer behavior
Date Sat, 02 Feb 2013 00:30:12 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569304#comment-13569304
] 

Jay Kreps commented on KAFKA-736:
---------------------------------

It would be nice to have an "ordering" torture test. There are actually lots of ways ordering
can get messed up (producer client code, network layer, broker, synchronization) and this
would help check that invariant. This can't really be done in a unit test very well since
it depends on load. This would be a command line tool that produced N messages from M threads.
Each message would be of the form T-i where T is the thread number and i is a sequential counter
incremented by the producer. When all N messages had been produced the tool would start a
consumer and consume all the messages and keep a map of threadId => last_seen. The last_seen
value should increase sequentially for each thread with no gaps.
                
> Add an option to the 0.8 producer to mimic 0.7 producer behavior
> ----------------------------------------------------------------
>
>                 Key: KAFKA-736
>                 URL: https://issues.apache.org/jira/browse/KAFKA-736
>             Project: Kafka
>          Issue Type: Improvement
>          Components: producer 
>    Affects Versions: 0.8
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>            Priority: Blocker
>              Labels: p2, replication-performance
>         Attachments: kafka-736-draft.patch, kafka-736-v1.patch, kafka-736-v2.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> I profiled a producer throughput benchmark between a producer and a remote broker. It
turns out that the background send threads spends ~97% of its time waiting to read the acknowledgement
from the broker.
> I propose we change the current behavior of request.required.acks=0 to mean no acknowledgement
from the broker. This will mimic the 0.7 producer behavior and will enable tuning the producer
for very high throughput.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message