flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Kafka offset commits
Date Wed, 19 Apr 2017 09:14:26 GMT
Hi,
AFAIK, restoring a Flink 1.0 savepoint should not be possible on Flink 1.2. Only restoring
from Flink 1.1 savepoints is supported.

@Gordon If the consumer group stays the same the new Flink job should pick up where the old
one stopped, right?

Best,
Aljoscha

> On 18. Apr 2017, at 16:19, Gwenhael Pasquiers <gwenhael.pasquiers@ericsson.com>
wrote:
> 
> Thanks for your answer.
> Does that means that flink does not rely on the offset in written to zookeeper anymore,
but relies on the snapshots data, implying that it’s crucial to keep the same snapshot folder
before and after the migration to Flink 1.2 ?
> We were also wondering if the flink consumer was able to restore it’s offset from Zookeeper.
> Another question : is there an expiration to the snapshots ? We’ve been having issues
with an app that we forgot to restart. We did it after a couple of days, but it looks like
it did not restore correctly the offset and it started consuming from the oldest offset, creating
duplicated data (the kafka queue has over a week of buffer).
> B.R.
>  
> From: Tzu-Li (Gordon) Tai [mailto:tzulitai@apache.org] 
> Sent: lundi 17 avril 2017 07:40
> To: user@flink.apache.org
> Subject: Re: Kafka offset commits
>  
> Hi,
>  
> The FlinkKafkaConsumer in 1.2 is able to restore from older version state snapshots and
bridge the migration, so there should be no problem in reading the offsets from older state.
The smallest or highest offsets will only be used if the offset no longer exists due to Kafka
data retention settings.
>  
> Besides this, there was once fix related to the Kafka 0.8 offsets for Flink 1.2.0 [1].
Shortly put, before the fix, the committed offsets to ZK was off by 1 (wrt to how Kafka itself
defines the committed offsets).
> However, this should not affect the behavior of restoring from offsets in savepoints,
so it should be fine.
>  
> Cheers,
> Gordon
>  
> [1] https://issues.apache.org/jira/browse/FLINK-4723 <https://issues.apache.org/jira/browse/FLINK-4723>
>  
> On 13 April 2017 at 10:55:40 PM, Gwenhael Pasquiers (gwenhael.pasquiers@ericsson.com
<mailto:gwenhael.pasquiers@ericsson.com>) wrote:
> 
> Hello,
>  
> We’re going to migrate some applications that consume data from a Kafka 0.8 from Flink
1.0 to Flink 1.2.
>  
> We are wondering if the offset commitment system changed between those two versions:
is there a risk that the Flink 1.2-based application will start with no offset (thus will
use either the smallest or highest one) ?
> Or can we assume that the Flink 1.2 app will resume its work at the same offset than
the Flink 1.0 app stopped (if they use the same consumer group id) ?
>  
> B.R.


Mime
View raw message