flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4514) ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream
Date Mon, 29 Aug 2016 13:16:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15445850#comment-15445850

ASF GitHub Bot commented on FLINK-4514:

Github user tzulitai commented on the issue:

    I think it'll also make sense to limit the config setting `ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS`
to be lower than the shard iterator expire time, otherwise the shard iterator will definitely
timeout on the next `getRecords()`.
    AWS documentation says the expire is 5 minutes (http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html),
I propose to set the limit to be 4.5 min, although I don't expect any user would actually
set such a high value.
    Adding this now...

> ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of stream
> --------------------------------------------------------------------------------
>                 Key: FLINK-4514
>                 URL: https://issues.apache.org/jira/browse/FLINK-4514
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.1.0, 1.1.1
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>             Fix For: 1.2.0, 1.1.2
> Original mailing thread for the reported issue:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html
> Normally, the exception is thrown when the consumer uses the same shard iterator after
5 minutes since it was retrieved. I've still yet to clarify & reproduce the root cause
of the {{ExpiredIteratorException}}, because from the code this seems to be impossible. I'm
leaning towards suspecting this is a Kinesis-side issue (from the description in the ML, the
behaviour also seems indeterminate).
> Either way, the exception can be fairly easily handled so that the consumer doesn't just
fail. When caught, we request a new shard iterator from Kinesis with the last sequence number.

This message was sent by Atlassian JIRA

View raw message