flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janardhan Reddy <janardhan.re...@olacabs.com>
Subject Re: Reading from latest offset in kafka consumer on restart
Date Wed, 03 Aug 2016 09:48:52 GMT
thanks.
We are using kafka flink consumer 0.8.2_11 ,I have set "auto.offset.reset"
to "largest"
On cancel and restart the consumer is reading from where it left off
instead of current offset, i tried both largest and latest in
auto.offset.reset



On Wed, Aug 3, 2016 at 3:12 PM, Stephan Ewen <sewen@apache.org> wrote:

> Checkpointing starts the consumer where it left off in case the job fails
> and recovers.
> If you explicitly cancel a job and start a new job (same jar), the new job
> will not start from a checkpoint, but from blank state.
>
>
> On Wed, Aug 3, 2016 at 11:37 AM, Janardhan Reddy <
> janardhan.reddy@olacabs.com> wrote:
>
>> I mean in case of chekpointing, won't the consumer start from where it
>> previously left ?
>>
>> On Wed, Aug 3, 2016 at 3:06 PM, Janardhan Reddy <
>> janardhan.reddy@olacabs.com> wrote:
>>
>>> How would checkpointing affect the offset.
>>>
>>> On Wed, Aug 3, 2016 at 3:03 PM, Stephan Ewen <sewen@apache.org> wrote:
>>>
>>>> When you cancel and restart a Flink job (without a savepoint), it does
>>>> not use the checkpoint data, and uses the behavior you defined in the Kafka
>>>> consumer to decide where to start from (consumer group, latest, or
>>>> earliest).
>>>>
>>>> On Wed, Aug 3, 2016 at 11:26 AM, Janardhan Reddy <
>>>> janardhan.reddy@olacabs.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Is there a way to read from latest offset in kafka consumer on
>>>>> restart.
>>>>> Or can we somehow start flink ignoring previous checkpointed data.
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message