hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leitao Guo (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7692) BlockPoolSliceStorage#loadBpStorageDirectories(...) should support MultiThread to speedup the upgrade of block pool at multi storage directories.
Date Wed, 28 Jan 2015 11:04:35 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Leitao Guo updated HDFS-7692:
-----------------------------
    Description: 
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
        if (containsStorageDir(dataDir)) {
          throw new IOException(
              "BlockPoolSliceStorage.recoverTransitionRead: " +
                  "attempt to load an used block storage: " + dataDir);
        }
        StorageDirectory sd =
            loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
        succeedDirs.add(sd);
      }
{code}

In the above code the storage directories will be analyzed one by one, which is really time
consuming when upgrading HDFS with datanodes have dozens of large volumes.  MultiThread dataDirs
analyzing should be supported here to speedup upgrade.

  was:
{code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
for (File dataDir : dataDirs) {
        if (containsStorageDir(dataDir)) {
          throw new IOException(
              "BlockPoolSliceStorage.recoverTransitionRead: " +
                  "attempt to load an used block storage: " + dataDir);
        }
        StorageDirectory sd =
            loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
        succeedDirs.add(sd);
      }
{code}

In the above code the storage directories will be analyzed one by one, which is really time
consuming when upgrading HDFS with datanodes have dozens of large volumes.  


> BlockPoolSliceStorage#loadBpStorageDirectories(...) should support MultiThread to speedup
the upgrade of block pool at multi storage directories.
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7692
>                 URL: https://issues.apache.org/jira/browse/HDFS-7692
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.5.2
>            Reporter: Leitao Guo
>
> {code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
> for (File dataDir : dataDirs) {
>         if (containsStorageDir(dataDir)) {
>           throw new IOException(
>               "BlockPoolSliceStorage.recoverTransitionRead: " +
>                   "attempt to load an used block storage: " + dataDir);
>         }
>         StorageDirectory sd =
>             loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
>         succeedDirs.add(sd);
>       }
> {code}
> In the above code the storage directories will be analyzed one by one, which is really
time consuming when upgrading HDFS with datanodes have dozens of large volumes.  MultiThread
dataDirs analyzing should be supported here to speedup upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message