kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: SimpleConsumer - Should FetchRequests span log segments?
Date Sat, 29 Jun 2013 15:42:46 GMT
When you iterate the fetch result, MessageAndOffset has a nextOffet()
method. You need to use that offset to initiate the next fetch request. See
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Examplefor
an example.

Thanks,

Jun


On Fri, Jun 28, 2013 at 9:53 PM, Ran RanUser <ranuser99@gmail.com> wrote:

> Using Kafka 0.8, when specifying a starting offset and appropriate
> fetchSize, SimpleConsumer will only return up to the highest offset of the
> log segment containing the starting offset.
>
> For example,
>
> log segment #1 contains offsets 1 - 10
> log segment #2 contiains offsets 11 - 100
>
> A fetch request specifying offset 9, will only return offsets 9 and 10.
> Specifying starting offset 11 will return 11 - 100
>
> Must one check fetchResponse.highwatermark each time to determine if there
> are higher numbered offsets, and continue submitting fetchRequests ?
>
> I assumed SimpleConsumer would return offsets/messages spanning log
> segments, up to the specified fetchSize.
>
>
> Thanks!
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message