flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tzuli...@apache.org
Subject [4/4] flink git commit: [FLINK-6365] [kinesis] Adapt default values of the Kinesis connector
Date Mon, 24 Jul 2017 03:55:17 GMT
[FLINK-6365] [kinesis] Adapt default values of the Kinesis connector

The previous GET_SHARDS_MAX and GET_SHARDS_INTERVAL_MILLIS did not work
well with AWS's service limitations, leading to poor Kinesis connector
performace if used directly out-of-the-box. This commit adapats them to
follow the default values used by the AWS SDK.

This closes #4375.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/35564f25
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/35564f25
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/35564f25

Branch: refs/heads/master
Commit: 35564f25c844b827ce325453b5d518416e1bd5a8
Parents: e16e9d9
Author: Bowen Li <bowenli86@gmail.com>
Authored: Wed Jul 19 23:35:28 2017 -0700
Committer: Tzu-Li (Gordon) Tai <tzulitai@apache.org>
Committed: Mon Jul 24 10:36:41 2017 +0800

----------------------------------------------------------------------
 docs/dev/connectors/kinesis.md                                   | 4 ++--
 .../connectors/kinesis/config/ConsumerConfigConstants.java       | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/35564f25/docs/dev/connectors/kinesis.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/kinesis.md b/docs/dev/connectors/kinesis.md
index 1fcc529..5fbf24b 100644
--- a/docs/dev/connectors/kinesis.md
+++ b/docs/dev/connectors/kinesis.md
@@ -250,8 +250,8 @@ of this API, the consumer will retry if Kinesis complains that the data
size / t
 up to a default of 3 attempts. Users can either try to slow down other non-Flink consuming
applications, or adjust the throughput
 of the consumer by setting the `ConsumerConfigConstants.SHARD_GETRECORDS_MAX` and
 `ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS` keys in the supplied configuration
properties. Setting the former
-adjusts the maximum number of records each consuming thread tries to fetch from shards on
each call (default is 100), while
-the latter modifies the sleep interval between each fetch (there will be no sleep by default).
The retry behaviour of the
+adjusts the maximum number of records each consuming thread tries to fetch from shards on
each call (default is 10,000), while
+the latter modifies the sleep interval between each fetch (default is 200). The retry behaviour
of the
 consumer when calling this API can also be modified by using the other keys prefixed by `ConsumerConfigConstants.SHARD_GETRECORDS_*`.
 
 ## Kinesis Producer

http://git-wip-us.apache.org/repos/asf/flink/blob/35564f25/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java
----------------------------------------------------------------------
diff --git a/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java
b/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java
index 8362776..702ed27 100644
--- a/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java
+++ b/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ConsumerConfigConstants.java
@@ -119,7 +119,7 @@ public class ConsumerConfigConstants extends AWSConfigConstants {
 
 	public static final double DEFAULT_STREAM_DESCRIBE_BACKOFF_EXPONENTIAL_CONSTANT = 1.5;
 
-	public static final int DEFAULT_SHARD_GETRECORDS_MAX = 100;
+	public static final int DEFAULT_SHARD_GETRECORDS_MAX = 10000;
 
 	public static final int DEFAULT_SHARD_GETRECORDS_RETRIES = 3;
 
@@ -129,7 +129,7 @@ public class ConsumerConfigConstants extends AWSConfigConstants {
 
 	public static final double DEFAULT_SHARD_GETRECORDS_BACKOFF_EXPONENTIAL_CONSTANT = 1.5;
 
-	public static final long DEFAULT_SHARD_GETRECORDS_INTERVAL_MILLIS = 0L;
+	public static final long DEFAULT_SHARD_GETRECORDS_INTERVAL_MILLIS = 200L;
 
 	public static final int DEFAULT_SHARD_GETITERATOR_RETRIES = 3;
 


Mime
View raw message