kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiangjie Qin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6028) Improve the quota throttle communication.
Date Wed, 11 Oct 2017 04:50:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199806#comment-16199806

Jiangjie Qin commented on KAFKA-6028:

[~junrao] We were thinking that the server will mute the channel for throttle time after return
the response with a throttle time. In that case, even if the clients sends the next request
immediately, the server won't read it from the socket until throttle time has passed. So a
bad client won't overwhelm the server. The only downside is that the receive socket buffer
will be used if a bad client does that. I have a KIP wiki ready and will post it shortly.

If we shrink the metric window the quota is essentially violated, right?

> Improve the quota throttle communication.
> -----------------------------------------
>                 Key: KAFKA-6028
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6028
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, core
>    Affects Versions: 1.0.0
>            Reporter: Jiangjie Qin
>            Assignee: Jiangjie Qin
>             Fix For: 1.1.0
> Currently if a client is throttled duet to quota violation, the broker will only send
back a response to the clients after the throttle time has passed. In this case, the clients
don't know how long the response will be throttled and might hit request timeout before the
response is returned. As a result the clients will retry sending a request and results a even
longer throttle time.
> The above scenario could happen when a large clients group sending records to the brokers.
We saw this when a MapReduce job pushes data to the Kafka cluster.
> To improve this, the broker can return the response with throttle time immediately after
processing the requests. After that, the broker will mute the channel for this client. A correct
client implementation should back off for that long before sending the next request. If the
client ignored the throttle time and send the next request immediately, the channel will be
muted and the request won't be processed until the throttle time has passed.
> A KIP will follow with more details.

This message was sent by Atlassian JIRA

View raw message