kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6028) Improve the quota throttle communication.
Date Thu, 05 Apr 2018 23:19:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427730#comment-16427730

ASF GitHub Bot commented on KAFKA-6028:

jonlee2 opened a new pull request #4830: KAFKA-6028: Improve the quota throttle communication
URL: https://github.com/apache/kafka/pull/4830
   This implements KIP-219, where a broker return a response with throttle time on quota violation
immediately after processing the corresponding request. After the response is sent out, the
broker will keep the channel muted until the throttle time is over. Also, on receiving a response
with throttle time, client will also block outgoing communication to the broker for the specified
throttle time.
   In the original KIP, it was stated that "In order to let the clients know whether the received
response is already throttled or not, we will need to bump up all the related request version."
However, after further discussion with @lindong28 and @becketqin, it was decided to bump up
the protocol version of ApiVersionsRequest/Response only, which should be sufficient to detect
the throttling behavior of the brokers that each client communicate with. 
   Initial review was done by @lindong28. 
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

> Improve the quota throttle communication.
> -----------------------------------------
>                 Key: KAFKA-6028
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6028
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, core
>    Affects Versions: 1.0.0
>            Reporter: Jiangjie Qin
>            Assignee: Jiangjie Qin
>            Priority: Major
>             Fix For: 1.2.0
> Currently if a client is throttled duet to quota violation, the broker will only send
back a response to the clients after the throttle time has passed. In this case, the clients
don't know how long the response will be throttled and might hit request timeout before the
response is returned. As a result the clients will retry sending a request and results a even
longer throttle time.
> The above scenario could happen when a large clients group sending records to the brokers.
We saw this when a MapReduce job pushes data to the Kafka cluster.
> To improve this, the broker can return the response with throttle time immediately after
processing the requests. After that, the broker will mute the channel for this client. A correct
client implementation should back off for that long before sending the next request. If the
client ignored the throttle time and send the next request immediately, the channel will be
muted and the request won't be processed until the throttle time has passed.
> A KIP will follow with more details.

This message was sent by Atlassian JIRA

View raw message