flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7223) Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector
Date Mon, 14 Aug 2017 15:41:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125845#comment-16125845

Stephan Ewen commented on FLINK-7223:

[~phoenixjiangnan] Do you think it is worthwhile filing an issue at Amazon for that?
It seems crazy that the number of describe requests is so low and across all apps of one account.

> Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector
> ----------------------------------------------------------------------------
>                 Key: FLINK-7223
>                 URL: https://issues.apache.org/jira/browse/FLINK-7223
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kinesis Connector
>    Affects Versions: 1.3.0
>            Reporter: Bowen Li
>            Assignee: Bowen Li
>            Priority: Minor
>             Fix For: 1.4.0
> Background: {{DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS}} in {{org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants}}
is the default value for Flink to call Kinesis's {{describeStream()}} API.
> Problem: Right now, its value is 10,000millis (10sec), which is too short. We ran into
problems that Flink-kinesis-connector's call of {{describeStream()}} exceeds Kinesis rate
limit, and broken Flink taskmanager.
> According to http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html,

> "This operation has a limit of 10 transactions per second per account.". What it means
is that the 10transaction/account is a limit on a single organization's AWS account......:(
 We contacted AWS Support, and confirmed this. If you have more applications (either other
Flink apps or non-Flink apps) competing aggressively with your Flink app on this API, your
Flink app breaks. 
> I propose increasing the value DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS from 10,000millis(10sec)
to preferably 300,000 (5min). Or at least 60,000 (1min) if anyone has a solid reason arguing
that 5min is too long, 
> This is also related to https://issues.apache.org/jira/browse/FLINK-6365

This message was sent by Atlassian JIRA

View raw message