kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5642) Use async ZookeeperClient in Controller
Date Thu, 31 Aug 2017 07:11:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148549#comment-16148549
] 

ASF GitHub Bot commented on KAFKA-5642:
---------------------------------------

GitHub user onurkaraman opened a pull request:

    https://github.com/apache/kafka/pull/3765

    KAFKA-5642 [WIP]: Use async ZookeeperClient in Controller

    Synchronous zookeeper writes means that we wait an entire round trip before doing the
next write. These synchronous writes are happening at a per-partition granularity in several
places, so partition-heavy clusters suffer from the controller doing many sequential round
trips to zookeeper:
    - PartitionStateMachine.electLeaderForPartition updates leaderAndIsr in zookeeper on transition
to OnlinePartition. This gets triggered per-partition sequentially with synchronous writes
during controlled shutdown of the shutting down broker's replicas for which it is the leader.
    - ReplicaStateMachine updates leaderAndIsr in zookeeper on transition to OfflineReplica
when calling KafkaController.removeReplicaFromIsr. This gets triggered per-partition sequentially
with synchronous writes for failed or controlled shutdown brokers.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/onurkaraman/kafka KAFKA-5642

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/3765.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3765
    
----
commit 4b3e4e8b1d403bf817aeaa9196fb69a15b1a13a0
Author: Onur Karaman <okaraman@linkedin.com>
Date:   2017-07-27T18:29:35Z

    KAFKA-5642: Use async ZookeeperClient in Controller
    
    Synchronous zookeeper writes means that we wait an entire round trip before doing the
next write. These synchronous writes are happening at a per-partition granularity in several
places, so partition-heavy clusters suffer from the controller doing many sequential round
trips to zookeeper:
    - PartitionStateMachine.electLeaderForPartition updates leaderAndIsr in zookeeper on transition
to OnlinePartition. This gets triggered per-partition sequentially with synchronous writes
during controlled shutdown of the shutting down broker's replicas for which it is the leader.
    - ReplicaStateMachine updates leaderAndIsr in zookeeper on transition to OfflineReplica
when calling KafkaController.removeReplicaFromIsr. This gets triggered per-partition sequentially
with synchronous writes for failed or controlled shutdown brokers.

----


> Use async ZookeeperClient in Controller
> ---------------------------------------
>
>                 Key: KAFKA-5642
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5642
>             Project: Kafka
>          Issue Type: Sub-task
>            Reporter: Onur Karaman
>            Assignee: Onur Karaman
>
> Synchronous zookeeper writes means that we wait an entire round trip before doing the
next write. These synchronous writes are happening at a per-partition granularity in several
places, so partition-heavy clusters suffer from the controller doing many sequential round
trips to zookeeper.
> * PartitionStateMachine.electLeaderForPartition updates leaderAndIsr in zookeeper on
transition to OnlinePartition. This gets triggered per-partition sequentially with synchronous
writes during controlled shutdown of the shutting down broker's replicas for which it is the
leader.
> * ReplicaStateMachine updates leaderAndIsr in zookeeper on transition to OfflineReplica
when calling KafkaController.removeReplicaFromIsr. This gets triggered per-partition sequentially
with synchronous writes for failed or controlled shutdown brokers.
> KAFKA-5501 introduced an async ZookeeperClient that encourages pipelined requests to
zookeeper. We should replace ZkClient's usage with this client.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message