incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bao Thai Ngo <baothai...@gmail.com>
Subject Re: Retrieve Partition-Id from MessageStream
Date Thu, 08 Dec 2011 06:51:38 GMT
Hi,

As far as I understand you intends to use semantic partitioning to maintain
some kind of produced data. This allows you to partition stream of messages
with some semantic partition function based on some key in message.
At consumer side, you also want to map semantic partitions with host
(server) names in order to consume the data properly.

Zk-base consumer approach does a little help in your case and you should
take care of getting semantic partitions you are interest of.
I would suggest you look at SimpleConsumer but you should take care of load
balancing, failover, offset management, etc .. on your own.

Any thought? I hope kafka guys will give you a right answer :)

~Thai

On Thu, Dec 8, 2011 at 12:54 PM, nn chuong411 <nnchuong411@gmail.com> wrote:

> Ok, Here is my usecase:
>   -I use a group of producers to tail log for one MMO game, one producer
> for one server game
>   -Each producer will send messages to a particular partition under a
> shared topic (mean, one topic for one game)
>   -I use a static map table to map partition-id - server game name
>   -A group of consumers will consume log for this game, one consumer for
> at least one partition
>   -Number of partitions for one consumer will be dynamic changed for load
> balancing at consumer side
>   -Processing at consumer require a know server name (mean, partition id)
> for each message
>   -I don't wanna handle offset updating manually with Simple - Consumer,
> it risk my processing logic
> So, do you have any suggestion for me?
>
> On Thu, Dec 8, 2011 at 12:21 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > No. Each message stream can have messages from multiple partitions and
> this
> > list can change with every rebalancing operation. Returning this
> > information to the user is a little tricky, but can you describe how you
> > would like to use it ? This will require some thought, I don't know the
> > right answer.
> >
> > Thanks,
> > Neha
> >
> > On Wed, Dec 7, 2011 at 7:37 PM, nn chuong411 <nnchuong411@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >    I'm using ZooKeeperConsumerConnector to create a group of
> > > messageStream. But is there anyway to retrieve partitionId associated
> > with
> > > each messageStream?
> > > --
> > > Nguyen Nam Chuong
> > > Software Engineer - VNG Cooperation
> > > Ho Chi Minh City - Vietnam
> > >
> >
>
>
>
> --
> Nguyen Nam Chuong
> Software Engineer - VNG Cooperation
> Ho Chi Minh City - Vietnam
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message