incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Retrieve Partition-Id from MessageStream
Date Thu, 08 Dec 2011 15:29:13 GMT
The current design of the high level consumer is that all messages from a
single partition (whether it's semantically partitioned or not) are
guaranteed to be consumed by one consumer. However, you don't get to choose
which consumer gets which partition. The kafka consumer code makes that
decision. We could return partition id to the consumer. However, the
following are some of the alternatives.

1. You could encode some partition specific information in the message
itself.

2. You could publish to different topics, instead of different partitions.

Thanks,

Jun

On Wed, Dec 7, 2011 at 9:54 PM, nn chuong411 <nnchuong411@gmail.com> wrote:

> Ok, Here is my usecase:
>   -I use a group of producers to tail log for one MMO game, one producer
> for one server game
>   -Each producer will send messages to a particular partition under a
> shared topic (mean, one topic for one game)
>   -I use a static map table to map partition-id - server game name
>   -A group of consumers will consume log for this game, one consumer for
> at least one partition
>   -Number of partitions for one consumer will be dynamic changed for load
> balancing at consumer side
>   -Processing at consumer require a know server name (mean, partition id)
> for each message
>   -I don't wanna handle offset updating manually with Simple - Consumer,
> it risk my processing logic
> So, do you have any suggestion for me?
>
> On Thu, Dec 8, 2011 at 12:21 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > No. Each message stream can have messages from multiple partitions and
> this
> > list can change with every rebalancing operation. Returning this
> > information to the user is a little tricky, but can you describe how you
> > would like to use it ? This will require some thought, I don't know the
> > right answer.
> >
> > Thanks,
> > Neha
> >
> > On Wed, Dec 7, 2011 at 7:37 PM, nn chuong411 <nnchuong411@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >    I'm using ZooKeeperConsumerConnector to create a group of
> > > messageStream. But is there anyway to retrieve partitionId associated
> > with
> > > each messageStream?
> > > --
> > > Nguyen Nam Chuong
> > > Software Engineer - VNG Cooperation
> > > Ho Chi Minh City - Vietnam
> > >
> >
>
>
>
> --
> Nguyen Nam Chuong
> Software Engineer - VNG Cooperation
> Ho Chi Minh City - Vietnam
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message