Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Wed, 30 Aug 2017 08:51:00 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.13092333.1501803083000.157490.1504083060130@Atlassian.JIRA>
In-Reply-To: <JIRA.13092333.1501803083000@Atlassian.JIRA>
References: <JIRA.13092333.1501803083000@Atlassian.JIRA> <JIRA.13092333.1501803083782@jira-lw-us.apache.org>
Subject: [jira] [Commented] (FLINK-7366) Upgrade kinesis producer library in
 flink-connector-kinesis
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 30 Aug 2017 08:51:05 -0000


    [ https://issues.apache.org/jira/browse/FLINK-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146913#comment-16146913 ] 

ASF GitHub Bot commented on FLINK-7366:
---------------------------------------

Github user aljoscha commented on the issue:

    https://github.com/apache/flink/pull/4522
  
    @tzulitai Is currently on vacation but before he left he added a Backwards compatibility test for the Kinesis consumer. We also dropped compatibility with the Flink 1.2 Kinesis consumer so I think we can also update the consumer library. (I realised this is not completely related to this PR but I just noticed it now, sorry for that.)


> Upgrade kinesis producer library in flink-connector-kinesis
> -----------------------------------------------------------
>
>                 Key: FLINK-7366
>                 URL: https://issues.apache.org/jira/browse/FLINK-7366
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.3.2
>            Reporter: Bowen Li
>            Assignee: Bowen Li
>             Fix For: 1.4.0, 1.3.3
>
>
> We need to upgrade KPL and KCL to pick up the enhanced performance and stability for Flink to work better  with Kinesis. Upgrading KPL is specially necessary, because the KPL version Flink uses is old, and doesn't have good retry and error handling logic.
> *KPL:*
> flink-connector-kinesis currently uses kinesis-producer-library 0.10.2, which is released in Nov 2015 by AWS. It's old. It's the fourth release, and thus problematic. It doesn't even have good retry logic, therefore Flink fails really frequently (about every 10 mins as we observed) when Flink writes too fast to Kinesis and receives RateLimitExceededException, 
> Quotes from https://github.com/awslabs/amazon-kinesis-producer/issues/56, "*With the newer version of the KPL it uses the AWS C++ SDK which should offer additional retries.*" on Oct 2016. 0.12.5, the version we are upgrading to, is released in May 2017 and should have the enhanced retry logic.
> *KCL:*
> Upgrade KCL from 1.6.2 to 1.8.1
> *AWS SDK*
> from 1.10.71 to 1.11.171


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)