flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8944) Use ListShards for shard discovery in the flink kinesis connector
Date Tue, 15 May 2018 17:54:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476257#comment-16476257

ASF GitHub Bot commented on FLINK-8944:

Github user tweise commented on a diff in the pull request:

    --- Diff: flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxyTest.java
    @@ -97,8 +228,7 @@ public void testClientConfigOverride() {
     		AmazonKinesis kinesisClient = Whitebox.getInternalState(proxy, "kinesisClient");
     		ClientConfiguration clientConfiguration = Whitebox.getInternalState(kinesisClient,
    -			"clientConfiguration");
    +						"clientConfiguration");
    --- End diff --
    remove unnecessary change

> Use ListShards for shard discovery in the flink kinesis connector
> -----------------------------------------------------------------
>                 Key: FLINK-8944
>                 URL: https://issues.apache.org/jira/browse/FLINK-8944
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Kailash Hassan Dayanand
>            Priority: Minor
> Currently the DescribeStream AWS API used to get list of shards is has a restricted rate
limits on AWS. (5 requests per sec per account). This is problematic when running multiple
flink jobs all on same account since each subtasks calls the Describe Stream. Changing this
to ListShards will provide more flexibility on rate limits as ListShards has a 100 requests
per second per data stream limits.
> More details on the mailing list. https://goo.gl/mRXjKh

This message was sent by Atlassian JIRA

View raw message