flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10154) Make sure we always read at least one record in KinesisConnector
Date Thu, 16 Aug 2018 15:43:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582718#comment-16582718

ASF GitHub Bot commented on FLINK-10154:

asfgit closed pull request #6564: [FLINK-10154] [kinesis connector] Make sure we always read
at least one record in Kinesis Connector.
URL: https://github.com/apache/flink/pull/6564

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
index d698ecf2e9d..6de72783f1f 100644
--- a/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
+++ b/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
@@ -278,8 +278,8 @@ private int adaptRecordsToRead(long runLoopTimeNanos, int numRecords,
long recor
 			double loopFrequencyHz = 1000000000.0d / runLoopTimeNanos;
 			double bytesPerRead = KINESIS_SHARD_BYTES_PER_SECOND_LIMIT / loopFrequencyHz;
 			maxNumberOfRecordsPerFetch = (int) (bytesPerRead / averageRecordSizeBytes);
-			// Ensure the value is not more than 10000L
-			maxNumberOfRecordsPerFetch = Math.min(maxNumberOfRecordsPerFetch, ConsumerConfigConstants.DEFAULT_SHARD_GETRECORDS_MAX);
+			// Ensure the value is greater than 0 and not more than 10000L
+			maxNumberOfRecordsPerFetch = Math.max(1, Math.min(maxNumberOfRecordsPerFetch, ConsumerConfigConstants.DEFAULT_SHARD_GETRECORDS_MAX));
 		return maxNumberOfRecordsPerFetch;


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

> Make sure we always read at least one record in KinesisConnector
> ----------------------------------------------------------------
>                 Key: FLINK-10154
>                 URL: https://issues.apache.org/jira/browse/FLINK-10154
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.6.0
>            Reporter: Jamie Grier
>            Assignee: Jamie Grier
>            Priority: Minor
>              Labels: pull-request-available
> It's possible in some cases to request zero records from Kinesis in the Kinesis connector. 
This can happen when the "adpative reads" feature is enabled.

This message was sent by Atlassian JIRA

View raw message