flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Stoyak <vsto...@yahoo.com>
Subject Re: Read Kafka topic from the beginning
Date Thu, 03 Dec 2015 18:06:13 GMT
Gave it a try, but does not seem to help. Is it working for you?

Thanks

Sent from my iPhone

> On Dec 3, 2015, at 6:11 PM, Vladimir Stoyak <vstoyak@yahoo.com> wrote:
> 
> As far as I know "auto.offset.reset" what to do if offset it not available or out of
bound?
> 
> Vladimir
> 
> 
> On Thursday, December 3, 2015 5:58 PM, Maximilian Michels <mxm@apache.org> wrote:
> Hi Vladimir,
> 
> You may supply Kafka consumer properties when you create the FlinkKafkaConsumer.
> 
> Properties props = new Properties();
> 
> // start from largest offset - DEFAULT
> props.setProperty("auto.offset.reset", "largest");
> // start from smallest offset
> props.setProperty("auto.offset.reset", "smallest");
> 
> I don't think it is possible to start from a specific offset. The
> offset is only unique per partition. You could modify the offsets in
> the Zookeeper state but you really have to know what you're doing
> then.
> 
> Best regards,
> Max
> 
> 
> 
>> On Thu, Dec 3, 2015 at 4:01 PM, Vladimir Stoyak <vstoyak@yahoo.com> wrote:
>> I see that Flink 0.10.1 now supports Keyed Schemas which allows us to rely on Kafka
topics set to "compact" retention for data persistence.
>> 
>> In our topology we wanted to set some topics with Log Compactions enabled and read
topic from the beginning when the topology starts or component recovers. Does current Kafka
Consumer implementation allow to read all messages in a topic from the beginning or from a
specific offset.
>> 
>> Thanks,
>> Vladimir

Mime
View raw message