Return-Path: X-Original-To: apmail-kafka-dev-archive@www.apache.org Delivered-To: apmail-kafka-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 955E3FFA9 for ; Wed, 3 Apr 2013 22:49:15 +0000 (UTC) Received: (qmail 70322 invoked by uid 500); 3 Apr 2013 22:49:15 -0000 Delivered-To: apmail-kafka-dev-archive@kafka.apache.org Received: (qmail 70292 invoked by uid 500); 3 Apr 2013 22:49:15 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 70282 invoked by uid 99); 3 Apr 2013 22:49:15 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Apr 2013 22:49:15 +0000 Date: Wed, 3 Apr 2013 22:49:15 +0000 (UTC) From: "Neha Narkhede (JIRA)" To: dev@kafka.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (KAFKA-330) Add delete topic support MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/KAFKA-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621456#comment-13621456 ] Neha Narkhede commented on KAFKA-330: ------------------------------------- Let's do some zookeeper math here to see how long it takes to delete one topic, 8 partitions from a 6 node kafka cluster - # of zk ops operation during delete topic 1 val partitionAssignment = ZkUtils.getPartitionAssignmentForTopics(zkClient, topics.toSeq) 7 val brokers = ZkUtils.getAllBrokersInCluster(zkClient) 1 ZkUtils.getAllReplicasOnBroker(zkClient, topics.toSeq, brokers.map(_.id)) (This is a redundant read from zookeeper, so reuse the info read in step 1) 2 removeReplicaFromIsr -> getLeaderIsrAndEpochForPartition, conditionalUpdatePersistentPath 9 removeFromTopicsBeingDeleted -> readDataMaybeNull (1), deletePath (8) 20 zookeeper ops. With 10ms per op, (which is a what a zookeeper cluster that kafka consumers and brokers share does in best case), that is 200ms per topic With 50 such topics, it is 10 seconds. That is the amount of time you are starving other partitions from being available! Also, the log side of your patch does not acquire the lock. You used the delete APIs that were used by unit tests so far. So they don't deal with the issues I've mentioned above in my comments. Regarding LogManager - Let's look at the modified version of your patch and see if that solves the problems I've outlined above wrt to interleaving other operations with delete log. > Add delete topic support > ------------------------- > > Key: KAFKA-330 > URL: https://issues.apache.org/jira/browse/KAFKA-330 > Project: Kafka > Issue Type: Bug > Components: controller, log, replication > Affects Versions: 0.8 > Reporter: Neha Narkhede > Assignee: Swapnil Ghike > Priority: Blocker > Labels: features, kafka-0.8, p2, project > Fix For: 0.8 > > Attachments: kafka-330-v1.patch > > > One proposal of this API is here - https://cwiki.apache.org/confluence/display/KAFKA/Kafka+replication+detailed+design+V2#KafkareplicationdetaileddesignV2-Deletetopic -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira