Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AE93F17A04 for ; Tue, 19 May 2015 06:39:10 +0000 (UTC) Received: (qmail 47775 invoked by uid 500); 19 May 2015 06:39:10 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 47728 invoked by uid 500); 19 May 2015 06:39:10 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 47716 invoked by uid 99); 19 May 2015 06:39:10 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2015 06:39:10 +0000 Date: Tue, 19 May 2015 06:39:10 +0000 (UTC) From: "Leitao Guo (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549901#comment-14549901 ] Leitao Guo commented on HDFS-7692: ---------------------------------- [~eddyxu], thanks for your comments, please have a check of the new patch. 1.In DataStorage#recoverTransitionRead, log the InterruptedException and rethrow it as InterruptedIOException; 2.In TestDataStorage#testAddStorageDirectoreis, catch InterruptedException then let the test case fail; 3.The multithread in DataStorage#addStorageLocations() is for one specific namespace, so in TestDataStorage#testAddStorageDirectoreis my intention is creating one thread pool for each namespace. Not change here. 4.Re-phrase the parameter successVolumes. [~szetszwo],thanks for your comments, please have a check of the new patch. 1. InterruptedException re-thrown as InterruptedIOException; 2. I think it's a good idea to log the upgrade progress for each dir, but so far, we can not get the progress easily from the current api. Do you think it's necessary to file a new jira to follow this? > DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories. > ---------------------------------------------------------------------------------------------------------------------------------- > > Key: HDFS-7692 > URL: https://issues.apache.org/jira/browse/HDFS-7692 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Affects Versions: 2.5.2 > Reporter: Leitao Guo > Assignee: Leitao Guo > Labels: BB2015-05-TBR > Attachments: HDFS-7692.01.patch, HDFS-7692.02.patch > > > {code:title=DataStorage#addStorageLocations(...)|borderStyle=solid} > for (StorageLocation dataDir : dataDirs) { > File root = dataDir.getFile(); > ... ... > bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs, startOpt); > addBlockPoolStorage(bpid, bpStorage); > ... ... > successVolumes.add(dataDir); > } > {code} > In the above code the storage directories will be analyzed one by one, which is really time consuming when upgrading HDFS with datanodes have dozens of large volumes. MultiThread dataDirs analyzing should be supported here to speedup upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332)