kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lucas Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6630) Speed up the processing of StopReplicaResponse events on the controller
Date Thu, 08 Mar 2018 22:54:00 GMT
Lucas Wang created KAFKA-6630:
---------------------------------

             Summary: Speed up the processing of StopReplicaResponse events on the controller
                 Key: KAFKA-6630
                 URL: https://issues.apache.org/jira/browse/KAFKA-6630
             Project: Kafka
          Issue Type: Improvement
          Components: core
            Reporter: Lucas Wang
            Assignee: Lucas Wang


Problem Statement:
We find in a large cluster with many partition replicas, it takes a long time to successfully
delete a topic. 

Root cause:
Further analysis shows that for a topic with N replicas, the controller receives all the N
StopReplicaResponses from brokers within a short time, however sequentially handling all the
N TopicDeletionStopReplicaResponseReceived events one by one takes a long time.

Specifically the functions triggered while handling every single TopicDeletionStopReplicaResponseReceived
event include:
TopicDeletionStopReplicaResponseReceived.process calls TopicDeletionManager.completeReplicaDeletion,
which calls TopicDeletionManager.resumeDeletions, which calls several inefficient functions.

The inefficient functions called inside TopicDeletionManager.resumeDeletions include
ReplicaStateMachine.areAllReplicasForTopicDeleted
ReplicaStateMachine.isAtLeastOneReplicaInDeletionStartedState
ReplicaStateMachine.replicasInState

Each of the 3 inefficient functions above will iterate through all the replicas in the cluster,
and filter out the replicas belonging to a topic. In a large cluster with many replicas, these
functions can be quite slow. 

Total deletion time for a topic becomes long in single threaded controller processing model:
Since the controller needs to sequentially process the queued TopicDeletionStopReplicaResponseReceived
events, if the time cost to process one event is t, the total time to process all events for
all replicas of a topic is N * t.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message