hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-270) DFS Upgrade should process dfs.data.dirs in parallel
Date Thu, 07 Oct 2010 22:32:34 GMT

     [ https://issues.apache.org/jira/browse/HDFS-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matt Foley updated HDFS-270:
----------------------------

          Component/s: data-node
    Affects Version/s:     (was: 0.22.0)
                       0.20.2
                 Tags: datanode startup, volume parallel
        Fix Version/s: 0.22.0

The principle issue with datanode upgrade speed turned out to be the per-file outcall to "exec",
see HDFS-1445.
However, running the upgrade volume-parallel is still very worthwhile, especially as we are
moving to 12-disk standard nodes.
Placing this work item under the umbrella bug HDFS-1443.

> DFS Upgrade should process dfs.data.dirs in parallel
> ----------------------------------------------------
>
>                 Key: HDFS-270
>                 URL: https://issues.apache.org/jira/browse/HDFS-270
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: 0.20.2
>            Reporter: Stu Hood
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>
> I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a little slowly.
> The main reason the upgrade took so long was the block upgrades on the datanodes. Each
of our datanodes has 3 drives listed for the dfs.data.dir parameter. From looking at the logs,
it is fairly clear that the upgrade procedure does not attempt to upgrade all listed dfs.data.dir's
in parallel.
> I think even if all of your dfs.data.dir's are on the same physical device, there would
still be an advantage to performing the upgrade process in parallel. The less downtime, the
better: especially if it is potentially 20 minutes versus 60 minutes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message