kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5828) Allow explicitly setting polling interval for Kafka connectors
Date Sun, 03 Sep 2017 21:00:02 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151950#comment-16151950

Richard Yu commented on KAFKA-5828:

In the Fetcher class, there is this {fetchRecord(int maxRecords)} method. It is indirectly
called from the {pollOnce()} method
in ConsumerConfigs. Using the above method, it should be relatively simple to set a higher

> Allow explicitly setting polling interval for Kafka connectors
> --------------------------------------------------------------
>                 Key: KAFKA-5828
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5828
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Behrang Saeedzadeh
> I have a Kafka cluster deployed in our internal data center and a Kafka Connect server
deployed in AWS that gets data records from a topic on the Kafka cluster and writes them into
a Kinesis stream.
> We want to ensure that our Kafka connect server does not saturate the available bandwidth
between our internal data center and AWS.
> Using {{max.partition.fetch.bytes}} we can limit the upper bound of data that can be
fetched in each poll call. If we can also configure the poll interval, then we can limit how
much data is transferred per partition per second.

This message was sent by Atlassian JIRA

View raw message