kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neha Narkhede (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-330) Add delete topic support
Date Wed, 03 Apr 2013 22:49:15 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621456#comment-13621456
] 

Neha Narkhede commented on KAFKA-330:
-------------------------------------

Let's do some zookeeper math here to see how long it takes to delete one topic, 8 partitions
from a 6 node kafka cluster -

# of zk ops                  operation during delete topic
1                                    val partitionAssignment = ZkUtils.getPartitionAssignmentForTopics(zkClient,
topics.toSeq)        
7                                    val brokers = ZkUtils.getAllBrokersInCluster(zkClient)
									
1                                    ZkUtils.getAllReplicasOnBroker(zkClient, topics.toSeq,
brokers.map(_.id))	        (This is a redundant read from zookeeper, so reuse the info read
in step 1)
2                                    removeReplicaFromIsr -> getLeaderIsrAndEpochForPartition,
conditionalUpdatePersistentPath
9                                    removeFromTopicsBeingDeleted -> readDataMaybeNull
(1), deletePath (8)

20 zookeeper ops. With 10ms per op, (which is a what a zookeeper cluster that kafka consumers
and brokers share does in best case), that is 200ms per topic

With 50 such topics, it is 10 seconds. That is the amount of time you are starving other partitions
from being available! 

Also, the log side of your patch does not acquire the lock. You used the delete APIs that
were used by unit tests so far. So they don't deal with the issues I've mentioned above in
my comments.

Regarding LogManager - Let's look at the modified version of your patch and see if that solves
the problems I've outlined above wrt to interleaving other operations with delete log.

                
> Add delete topic support 
> -------------------------
>
>                 Key: KAFKA-330
>                 URL: https://issues.apache.org/jira/browse/KAFKA-330
>             Project: Kafka
>          Issue Type: Bug
>          Components: controller, log, replication
>    Affects Versions: 0.8
>            Reporter: Neha Narkhede
>            Assignee: Swapnil Ghike
>            Priority: Blocker
>              Labels: features, kafka-0.8, p2, project
>             Fix For: 0.8
>
>         Attachments: kafka-330-v1.patch
>
>
> One proposal of this API is here - https://cwiki.apache.org/confluence/display/KAFKA/Kafka+replication+detailed+design+V2#KafkareplicationdetaileddesignV2-Deletetopic

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message