flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7367) Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections, RequestTimeout, etc)
Date Mon, 14 Aug 2017 06:17:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125289#comment-16125289
] 

ASF GitHub Bot commented on FLINK-7367:
---------------------------------------

Github user bowenli86 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/4473#discussion_r132879596
  
    --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisProducer.java
---
    @@ -57,6 +55,9 @@
     	/** Properties to parametrize settings such as AWS service region, access key etc. */
     	private final Properties configProps;
     
    +	/** Configuration for KinesisProducer. */
    +	private final KinesisProducerConfiguration producerConfig;
    --- End diff --
    
    agree. I'm moving it to `open()`


> Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections,
RequestTimeout, etc)
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-7367
>                 URL: https://issues.apache.org/jira/browse/FLINK-7367
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.3.0
>            Reporter: Bowen Li
>            Assignee: Bowen Li
>             Fix For: 1.4.0, 1.3.3
>
>
> Right now, FlinkKinesisProducer only expose two configs for the underlying KinesisProducer:
> - AGGREGATION_MAX_COUNT
> - COLLECTION_MAX_COUNT
> Well, according to [AWS doc|http://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html]
and [their sample on github|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties],
developers can set more to make the max use of KinesisProducer, and make it fault-tolerant
(e.g. by increasing timeout).
> I select a few more configs that we need when using Flink with Kinesis:
> - MAX_CONNECTIONS
> - RATE_LIMIT
> - RECORD_MAX_BUFFERED_TIME
> - RECORD_TIME_TO_LIVE
> - REQUEST_TIMEOUT
> Flink is using KPL's default values. They make Flink writing too fast to Kinesis, which
fail Flink job too frequently. We need to parameterize FlinkKinesisProducer to pass in the
above params, in order to slowing down Flink's write rate to Kinesis.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message