kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hussain Pirosha <hussain.piro...@impetus.co.in>
Subject RE: How to get partition information from Kafka Stream?
Date Fri, 28 Jun 2013 05:45:07 GMT
Great Suggestion !

Hope creating too many topics does not have any performance impact on the broker. Will it
?

Thanks,
Hussain

-----Original Message-----
From: Philip O'Toole [mailto:philip@loggly.com]
Sent: Thursday, June 27, 2013 9:03 PM
To: users@kafka.apache.org
Subject: Re: How to get partition information from Kafka Stream?

Could you just use a topic for each stage? The nice thing about topics is you don't even have
to tell the Kafka cluster about the topics beforehand, so if you a new stage appears, just
start writing to that topic.

Kafka is great. :-)

Phiilp

On Thu, Jun 27, 2013 at 7:27 AM, Hussain Pirosha <hussain.pirosha@impetus.co.in> wrote:
> A source produces data in kafka at different stages of execution. Topic is partitioned
by number of execution stages.
> Partition number shall give us that information.
>
> Eg. 0 specifies data received by source, #last partition specifies processed data delivered
by source etc.
>
> Thanks,
> Hussain
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, June 27, 2013 7:47 PM
> To: users@kafka.apache.org
> Subject: Re: How to get partition information from Kafka Stream?
>
> Currently, partition # is not exposed in MessageAndMetaData. We do plan to expose such
information in a future release. How do you plan to use this info?
>
> Thanks,
>
> Jun
>
>
> On Thu, Jun 27, 2013 at 7:13 AM, Hussain Pirosha < hussain.pirosha@impetus.co.in>
wrote:
>
>> After retrieving a kafka stream or kafka message how to get the
>> corresponding partition number to which it belongs ? I am using kafka
>> version 0.7.2.
>> More specifically kafka.consumer.KafkaStream and
>> kafka.message.MessageAndMetaData classes, does not provide API to
>> retrieve partition number. Are there any other API's to get the partition number
?
>>
>> Thanks,
>> Hussain
>>
>>
>> ________________________________
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited
>> when received in error. Impetus does not represent, warrant and/or
>> guarantee, that the integrity of this communication has been
>> maintained nor that the communication is free of errors, virus, interception or interference.
>>
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential, proprietary, privileged
or otherwise protected by law. The message is intended solely for the named addressee. If
received in error, please destroy and notify the sender. Any use of this email is prohibited
when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity
of this communication has been maintained nor that the communication is free of errors, virus,
interception or interference.

________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

Mime
View raw message