kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: How to always consume from latest offset in kafka-streams
Date Fri, 19 Jan 2018 23:13:39 GMT
That is not supported out-of-box.

Configuration "auto.offset.reset" only triggers, if there are not
committed offsets and there is KS config to change this behavior.

A possible workaround might be (but not sure if I want to recommend
this), to increase KafkaStreams commit interval via StreamsConfig (you
could set it to MAX_VALUE, effectively disable committing). Thus,
auto.offset.reset should trigger on restart. You might want to try it
out and see if it works for your... Note, if we never commit, we also
never flush KTable caches, thus you might need to disable caching, too
(by setting cache size to zero).

As an alternative, you could manipulate offsets manually before startup
using bin/kafka-consumer-groups.sh --- the application.id is the
group.id and and you could "seek to end" before you restart the application.

Hope this helps.


-Matthias

On 1/19/18 9:22 AM, Saloni Vithalani wrote:
> Our requirement is such that if a kafka-stream app is consuming a
> partition, it should start it's consumption from latest offset of that
> partition.
> 
> This seems like do-able using
> 
> streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")
> 
> Now, let's say using above configuration, the kafka-stream app started
> consuming data from latest offset for a partition. And after some time, the
> app crashes. When the app comes back live, we want it to consume data from
> the latest offset of that partition, instead of the where it left last
> reading.
> 
> But I can't find anything that can help achieve it using kafka-streams api.
> 
> P.S. We are using kafka-1.0.0.
> 
> Saloni Vithalani
> Developer
> Email saloniv@thoughtworks.com
> Telephone +91 8552889571 <8552889571>
> [image: ThoughtWorks]
> <http://www.thoughtworks.com/?utm_campaign=saloni-vithalani-signature&utm_medium=email&utm_source=thoughtworks-email-signature-generator>
> 


Mime
View raw message