incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <>
Subject Re: understanding partitions based on wiki example of profile visits
Date Mon, 26 Nov 2012 16:53:33 GMT
Yes, your description is correct. A particular member's data would all be
in one partition.

Broker partitions are just the unit of parallelism--think of each partition
as a totally ordered log you can append to and read from. The consumption
of one of these partition logs is single threaded.

The guarantee is that all messages are added to a partition in the order
they arrive. From the point of view of a single producer client this will
also be the order in which they are sent. These messages are then delivered
in this order to a consumer thread.

Hope that helps.


On Sun, Nov 25, 2012 at 7:54 PM, S Ahmed <> wrote:

> The wiki states "Consider an application that would like to maintain an
> aggregation of the number of profile visitors for each member. It would
> like to send all profile visit events for a member to a particular
> partition and, hence, have all updates for a member to appear in the same
> stream for the same consumer thread." (
> So say I have 5 broker servers, now my producer will send a message for a
> particular profile page visit, with the default algorithm using
> hash(member_id)%num_partitions
> to figur out which broker server to send it it.
> So a particular members pageview messages will all go to a single server
> then, is this the case?  And therefore all the messages for a given user
> will be in the correct order also right?
> So a consumer group that subscribes to the 'profile-page-view' topic will
> consume page view related messages, is it possible to subscribe to a
> particular broker partition also?
> Are broker partitions meant for cases when you want all messages to be
> saved on the same node?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message