kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Kreps (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (KAFKA-208) Efficient polling mechanism for many topics
Date Wed, 20 Feb 2013 17:29:13 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jay Kreps resolved KAFKA-208.
-----------------------------

    Resolution: Fixed

This is fixed in the new 0.8 protocol 
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
                
> Efficient polling mechanism for many topics
> -------------------------------------------
>
>                 Key: KAFKA-208
>                 URL: https://issues.apache.org/jira/browse/KAFKA-208
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Taylor Gautier
>
> Currently, the way to poll many topics is to submit a request for each one in turn, and
read the responses.  Especially if there are few messages delivered on many topics, the network
overhead to implement this scheme can far outweigh the bandwidth of the actual messages delivered.
> Effectively, for topics A, B, C the request/response scheme is the following:
> -> Request A offset a
> -> Request B offset b
> -> Request C offset c
> <- no messages
> <- 1 message offset b
> <- no messages
> -> Request A offset a
> -> Request B offset b'
> -> Request C offset c
> <- no messages
> <- no messages
> <- no messages
> etc.
> I propose a more efficient mechanism which works a bit like epoll in that the client
can register interest in a particular topic.  There are many options for the implementation,
but the one I suggest goes like so:
> -> Register interest A offset a
> -> Register interest B offset b
> -> Register interest C offset c
> -> Next message (null)
> <- messages for B (1 message)
> -> Next message (topic B offset b')
> <- no messages
> -> Unregister Interest C
> ...
> It is possible to implement the "Next Message" request as either blocking or non-blocking.
 I suggest that the request format include space for the timeout, which if set to 0 will be
a nonblocking response, and if set to anything other than 0, will block for at most timeout
ms. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message