flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bowen Li (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (FLINK-7223) Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector
Date Tue, 03 Oct 2017 22:17:01 GMT

     [ https://issues.apache.org/jira/browse/FLINK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Bowen Li resolved FLINK-7223.
    Resolution: Won't Fix

> Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector
> ----------------------------------------------------------------------------
>                 Key: FLINK-7223
>                 URL: https://issues.apache.org/jira/browse/FLINK-7223
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kinesis Connector
>    Affects Versions: 1.3.0
>            Reporter: Bowen Li
>            Assignee: Bowen Li
>            Priority: Minor
>             Fix For: 1.4.0
> Background: {{DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS}} in {{org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants}}
is the default value for Flink to call Kinesis's {{describeStream()}} API.
> Problem: Right now, its value is 10,000millis (10sec), which is too short. We ran into
problems that Flink-kinesis-connector's call of {{describeStream()}} exceeds Kinesis rate
limit, and broken Flink taskmanager.
> According to http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html,

> "This operation has a limit of 10 transactions per second per account.". What it means
is that the 10transaction/account is a limit on a single organization's AWS account......:(
 We contacted AWS Support, and confirmed this. If you have more applications (either other
Flink apps or non-Flink apps) competing aggressively with your Flink app on this API, your
Flink app breaks. 
> I propose increasing the value DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS from 10,000millis(10sec)
to preferably 300,000 (5min). Or at least 60,000 (1min) if anyone has a solid reason arguing
that 5min is too long, 
> This is also related to https://issues.apache.org/jira/browse/FLINK-6365

This message was sent by Atlassian JIRA

View raw message