kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie Wang <jamie.w...@actuate.com>
Subject RE: How do you keep track of offset in a partition
Date Tue, 29 Jan 2013 02:14:50 GMT
Thanks. This makes sense. 


-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com] 
Sent: Monday, January 28, 2013 5:27 PM
To: users@kafka.apache.org
Subject: Re: How do you keep track of offset in a partition


You need to use the getOffsetsBefore() API to get the earliest/latest
offset available on the broker for a particular partition.


On Mon, Jan 28, 2013 at 5:05 PM, Jamie Wang <jamie.wang@actuate.com> wrote:

> Hi,
> We are using 0.72 version of Kafka on Windows. I am wondering what is the
> right way to fetch data and keep track of offset in a partition. For
> example, I am currently assuming the first message the producer sent to the
> broker is at offset 0. So far it seems working. Is this correct assumption?
> Let' say 2 days later, the first 100 messages on the broker is discarded
> because it passed retention.hours set in the config file. Now what is the
> offset I should use to retrieve the first message in the partition?  And
> let's also say the offset I had for the 80th message is now not valid.
> What is the right way to get the correct offset to fetch in the consumer?
> What is the purpose of the api for getting a list of valid offsets for all
> segments in a partition?
> Thanks in advance for your help.
> Jamie

View raw message