flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh <jof...@gmail.com>
Subject Kinesis connector - Iterator expired exception
Date Fri, 26 Aug 2016 11:20:55 GMT
Hi all,

I guess this is probably a question for Gordon - I've been using the
Flink-Kinesis connector for a while now and seen this exception a couple of
times:

com.amazonaws.services.kinesis.model.ExpiredIteratorException:
Iterator expired. The iterator was created at time Fri Aug 26 10:47:47
UTC 2016 while right now it is Fri Aug 26 11:05:40 UTC 2016 which is
further in the future than the tolerated delay of 300000 milliseconds.
(Service: AmazonKinesis; Status Code: 400; Error Code:
ExpiredIteratorException; Request ID:
d3db1d90-df97-912b-83e1-3954e766bbe0)


It happens when my Flink job goes down for a couple of hours, then I
restart from the existing state and it needs to catch up on all the
data that has been put in Kinesis stream in the hours where the job
was down. The job then runs for ~15 mins and fails with this exception
(and this happens repeatedly - meaning I can't restore the job from
the existing state).


Any ideas what's causing this? It's possible that it's been fixed in
recent commits, as the version of the Kinesis connector I'm using is
behind master - I'm not sure exactly what commit I'm using (doh!) but
it was built around mid June.


Thanks,

Josh

Mime
View raw message