Mailing-List: contact jira-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@kafka.apache.org
Date: Sun, 3 Sep 2017 21:05:02 +0000 (UTC)
From: "Richard Yu (JIRA)" <jira@apache.org>
To: jira@kafka.apache.org
Message-ID: <JIRA.13099454.1504448343000.17981.1504472702063@Atlassian.JIRA>
In-Reply-To: <JIRA.13099454.1504448343000@Atlassian.JIRA>
References: <JIRA.13099454.1504448343000@Atlassian.JIRA> <JIRA.13099454.1504448343224@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (KAFKA-5828) Allow explicitly setting
 polling interval for Kafka connectors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 03 Sep 2017 21:05:13 -0000


    [ https://issues.apache.org/jira/browse/KAFKA-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151950#comment-16151950 ] 

Richard Yu edited comment on KAFKA-5828 at 9/3/17 9:05 PM:
-----------------------------------------------------------

In the Fetcher class, there is the {code} fetchRecord(int maxRecords) {code} method. It is indirectly called from the {code} pollOnce(long timeout) {code} method in ConsumerConfigs. Using the above method, it should be relatively simple to set a higher bound. Also, the {code} poll(long timeout) {code} method calls {code} pollOnce(long timeout) {code}. Using the timeout parameter, the interval could also be set.


was (Author: yohan123):
In the Fetcher class, there is the
 {code} fetchRecord(int maxRecords) {code} method. It is indirectly called from the {code} pollOnce(long timeout) {code} method in ConsumerConfigs. Using the above method, it should be relatively simple to set a higher bound.

Also, the {code} poll(long timeout) {code} method calls {code} pollOnce(long timeout) {code}. 
Using the timeout parameter, the interval could also be set.


> Allow explicitly setting polling interval for Kafka connectors
> --------------------------------------------------------------
>
>                 Key: KAFKA-5828
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5828
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Behrang Saeedzadeh
>
> I have a Kafka cluster deployed in our internal data center and a Kafka Connect server deployed in AWS that gets data records from a topic on the Kafka cluster and writes them into a Kinesis stream.
> We want to ensure that our Kafka connect server does not saturate the available bandwidth between our internal data center and AWS.
> Using {{max.partition.fetch.bytes}} we can limit the upper bound of data that can be fetched in each poll call. If we can also configure the poll interval, then we can limit how much data is transferred per partition per second.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)