flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steffen Hausmann (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-6365) Adapt default values of the Kinesis connector
Date Sun, 23 Apr 2017 19:24:04 GMT
Steffen Hausmann created FLINK-6365:
---------------------------------------

             Summary: Adapt default values of the Kinesis connector
                 Key: FLINK-6365
                 URL: https://issues.apache.org/jira/browse/FLINK-6365
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
    Affects Versions: 1.2.0
            Reporter: Steffen Hausmann
            Priority: Minor


As discussed in http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-SHARD-GETRECORDS-MAX-default-value-td12332.html,
it seems reasonable to change the default values of the Kinesis connector to follow KCL’s
default settings. I suggest to adapt at least the values for SHARD_GETRECORDS_MAX and
SHARD_GETRECORDS_INTERVAL_MILLIS. 

As a Kinesis shard is currently limited to 5 get operations per second, you can observe high
ReadProvisionedThroughputExceeded rates with the current default value for SHARD_GETRECORDS_INTERVAL_MILLIS
of 0; it seem reasonable to increase it to 200. As it's described in the email thread, it
seems furthermore desirable to increase the default value for SHARD_GETRECORDS_MAX to 10000.

The values that are used by the KCL can be found here: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java

Thanks for looking into this!

Steffen



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message