flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Encountered the following Expired Iterator exception in getRecords using FlinkKinesisConsumer
Date Thu, 13 Dec 2018 10:59:03 GMT

Thanks for reporting this.

This looks like an overlooked corner case that the Kinesis connector doesn’t handle properly.

First, let me clarify the case and how it can be reproduced. Please let me know if the following
is correct:
1. You started a Kinesis connector source, with TRIM_HORIZON as the startup position.
2. No records were written to the Kinesis stream at all.
3. After a period of time, you received the “Encountered an unexpected expired iterator”
warning in the logs, and the job failed with the misleading AmazonKinesisException?

On 13 December 2018 at 6:53:11 AM, Vijay Balakrishnan (bvijaykr@gmail.com) wrote:

Using FlinkKinesisConsumer in a long running Flink Streaming app consuming from a Kinesis
Encountered the following Expired Iterator exception in getRecords():
 org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer [] - Encountered an
unexpected expired iterator 
The error on the console ends up being a misleading one: "Caused by: org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.model.AmazonKinesisException:
1 validation error detected: Value 'EARLIEST_SEQUENCE_NUM' at 'startingSequenceNumber' failed
to satisfy constraint: Member must satisfy regular expression pattern: 0|([1-9]\d{0,128})
(Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException; Request ID: ..)
How do I increase the ClientConfiguration.clientExecutiontimeout to avoid this issue or is
this the right way to handle this issue ? I don't want the FlinkKinesisConsumer streaming
app to fail just because there might be no records in the Kinesis Stream. I am using TRIM_HORIZON
to read from the start of the Kinesis Stream.
View raw message