hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leitao Guo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7692) DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
Date Sat, 07 Feb 2015 07:00:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310587#comment-14310587
] 

Leitao Guo commented on HDFS-7692:
----------------------------------

[~eddyxu], thanks for your review and comments!

When running tests after patching, I got "NullPointerException" at this line, so I add the
check of "null != datanode.getConf()" here.
{code} Executors.newFixedThreadPool(null != datanode.getConf() ? datanode
493	            .getConf().getInt(
494	                DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_THREADS_KEY,
495	                DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT)
496	            : dataDirs.size()); {code}

I will update the patch according to your comments, thanks!

> DataStorage#addStorageLocations(...) should support MultiThread to speedup the upgrade
of block pool at multi storage directories.
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7692
>                 URL: https://issues.apache.org/jira/browse/HDFS-7692
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.5.2
>            Reporter: Leitao Guo
>            Assignee: Leitao Guo
>         Attachments: HDFS-7692.01.patch
>
>
> {code:title=DataStorage#addStorageLocations(...)|borderStyle=solid}
>     for (StorageLocation dataDir : dataDirs) {
>       File root = dataDir.getFile();
>      ... ...
>         bpStorage.recoverTransitionRead(datanode, nsInfo, bpDataDirs, startOpt);
>         addBlockPoolStorage(bpid, bpStorage);
>     ... ...
>       successVolumes.add(dataDir);
>     }
> {code}
> In the above code the storage directories will be analyzed one by one, which is really
time consuming when upgrading HDFS with datanodes have dozens of large volumes.  MultiThread
dataDirs analyzing should be supported here to speedup upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message