kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiangjie Qin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool
Date Fri, 29 Jan 2016 18:40:40 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123956#comment-15123956
] 

Jiangjie Qin commented on KAFKA-1464:
-------------------------------------

It looks our purpose is to minimize user impact during replica catching up. From broker point
of view, as long as client request latency is acceptable we should fully utilize the bandwidth
we have to let replicas keep up. We should be able to measure the user experience by checking
Queuing time of requests from and response to clients.

If that is the case, maybe we can let user set an SLA for latency. And we will not throttle
replication as long as the user ProduceRequest / FetchRequest queuing time. Otherwise, we
will throttle the fetching from out of sync replica (We probably don't want to throttle in-sync
replicas).

> Add a throttling option to the Kafka replication tool
> -----------------------------------------------------
>
>                 Key: KAFKA-1464
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1464
>             Project: Kafka
>          Issue Type: New Feature
>          Components: replication
>    Affects Versions: 0.8.0
>            Reporter: mjuarez
>            Assignee: Ismael Juma
>            Priority: Minor
>              Labels: replication, replication-tools
>             Fix For: 0.9.1.0
>
>
> When performing replication on new nodes of a Kafka cluster, the replication process
will use all available resources to replicate as fast as possible.  This causes performance
issues (mostly disk IO and sometimes network bandwidth) when doing this in a production environment,
in which you're trying to serve downstream applications, at the same time you're performing
maintenance on the Kafka cluster.
> An option to throttle the replication to a specific rate (in either MB/s or activities/second)
would help production systems to better handle maintenance tasks while still serving downstream
applications.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message