flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balaji Rajagopalan <balaji.rajagopa...@olacabs.com>
Subject Re: reading from latest kafka offset when flink starts
Date Fri, 06 May 2016 10:07:55 GMT
Thanks Robert appreciate your help.

On Fri, May 6, 2016 at 3:07 PM, Robert Metzger <rmetzger@apache.org> wrote:

> Hi,
>
> yes, you can use Kafka's configuration setting for that. Its called
> "auto.offset.reset". Setting it to "latest" will change the restart
> behavior to the current offset ("earliest" is the opposite).
>
> How heavy is the processing you are doing? 4500 events/second sounds not
> like a lot of throughput.
>
> On Wed, May 4, 2016 at 8:24 AM, Balaji Rajagopalan <
> balaji.rajagopalan@olacabs.com> wrote:
>
>> I am using the flink connector to read from a kafka stream, I ran into
>> the problem where the flink job went down due to some application error, it
>> was down for sometime, meanwhile the kafka queue was growing as expected no
>> consumer to consume from the given group , and when I started the flink it
>> started consuming the messages no problem so far, but consumer lag was huge
>> since producer is a fast producer about 4500 events/sec. My question is
>> there any flink connector configuration which can force it read from the
>> latest offset when the flink application starts since in my application
>> logic I do not care about older events.
>>
>> balaji
>>
>
>

Mime
View raw message