flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Kinesis connector SHARD_GETRECORDS_MAX default value
Date Wed, 22 Mar 2017 19:21:19 GMT
Hi Steffan,

I have to admit that I didn’t put too much thoughts in the default values for the Kinesis
consumer.

I’d say it would be reasonable to change the default values to follow KCL’s settings.
Could you file a JIRA for this?

In general, we might want to reconsider all the default values for configs related to the
getRecords call, i.e.
- SHARD_GETRECORDS_MAX
- SHARD_GETRECORDS_INTERVAL_MILLIS
- SHARD_GETRECORDS_BACKOFF_*

Cheers,
Gordon

On March 23, 2017 at 2:12:32 AM, Steffen Hausmann (steffen@hausmann-family.de) wrote:

Hi there,  

I recently ran into problems with a Flink job running on an EMR cluster  
consuming events from a Kinesis stream receiving roughly 15k  
event/second. Although the EMR cluster was substantially scaled and CPU  
utilization and system load were well below any alarming threshold, the  
processing of events of the stream increasingly fell behind.  

Eventually, it turned out that the SHARD_GETRECORDS_MAX defaults to 100  
which is apparently causing too much overhead when consuming events from  
the stream. Increasing the value to 5000, a single GetRecords call to  
Kinesis can retrieve up to 10k records, made the problem go away.  

I wonder why the default value for SHARD_GETRECORDS_MAX is chosen so low  
(100x less than it could be). The Kinesis Client Library defaults to  
5000 and it's recommended to use this default value:  
http://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html#consumer-app-reading-slower.
 

Thanks for the clarification!  

Cheers,  
Steffen  

Mime
View raw message