Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Fri, 11 Aug 2017 08:29:00 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.13092337.1501805130000.8899.1502440140463@Atlassian.JIRA>
In-Reply-To: <JIRA.13092337.1501805130000@Atlassian.JIRA>
References: <JIRA.13092337.1501805130000@Atlassian.JIRA> <JIRA.13092337.1501805130421@jira-lw-us.apache.org>
Subject: [jira] [Commented] (FLINK-7367) Parameterize more configs for
 FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections,
 RequestTimeout, etc)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 11 Aug 2017 08:29:05 -0000


    [ https://issues.apache.org/jira/browse/FLINK-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123030#comment-16123030 ] 

ASF GitHub Bot commented on FLINK-7367:
---------------------------------------

Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/4473#discussion_r132636379
  
    --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/ProducerConfigConstants.java ---
    @@ -22,12 +22,14 @@
     /**
      * Optional producer specific configuration keys for {@link FlinkKinesisProducer}.
      */
    +@Deprecated
     public class ProducerConfigConstants extends AWSConfigConstants {
     
    -	/** Maximum number of items to pack into an PutRecords request. **/
    -	public static final String COLLECTION_MAX_COUNT = "aws.producer.collectionMaxCount";
    -
    -	/** Maximum number of items to pack into an aggregated record. **/
    -	public static final String AGGREGATION_MAX_COUNT = "aws.producer.aggregationMaxCount";
    +	/** Deprecated key. **/
    +	@Deprecated
    +	public static final String DEPRECATED_COLLECTION_MAX_COUNT = "aws.producer.collectionMaxCount";
     
    + 	/** Deprecated key. **/
    +	@Deprecated
    +	public static final String DEPRECATED_AGGREGATION_MAX_COUNT = "aws.producer.aggregationMaxCount";
    --- End diff --
    
    This cannot be renamed, as it completely breaks user code that uses `ProducerConfigConstants. AGGREGATION_MAX_COUNT`.


> Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections, RequestTimeout, etc)
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-7367
>                 URL: https://issues.apache.org/jira/browse/FLINK-7367
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.3.0
>            Reporter: Bowen Li
>            Assignee: Bowen Li
>             Fix For: 1.4.0, 1.3.3
>
>
> Right now, FlinkKinesisProducer only expose two configs for the underlying KinesisProducer:
> - AGGREGATION_MAX_COUNT
> - COLLECTION_MAX_COUNT
> Well, according to [AWS doc|http://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html] and [their sample on github|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties], developers can set more to make the max use of KinesisProducer, and make it fault-tolerant (e.g. by increasing timeout).
> I select a few more configs that we need when using Flink with Kinesis:
> - MAX_CONNECTIONS
> - RATE_LIMIT
> - RECORD_MAX_BUFFERED_TIME
> - RECORD_TIME_TO_LIVE
> - REQUEST_TIMEOUT
> Flink is using KPL's default values. They make Flink writing too fast to Kinesis, which fail Flink job too frequently. We need to parameterize FlinkKinesisProducer to pass in the above params, in order to slowing down Flink's write rate to Kinesis.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)