hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-270) DFS Upgrade should process dfs.data.dirs in parallel
Date Wed, 09 Feb 2011 18:31:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992639#comment-12992639

Matt Foley commented on HDFS-270:

Hello Hairong, sorry I didn't see and respond to your request timely.
You are of course welcome to take this on.

However, please first take the patch for HDFS-1445, which I have now uploaded to that JIRA.
 It cuts the per-volume upgrade time from aprx 8 minutes to aprx 8 seconds, for my timings
of a 12,500-block (25,000-file) volume.  Even 12 volumes won't take very long to upgrade at
that rate.
Regards, --Matt

> DFS Upgrade should process dfs.data.dirs in parallel
> ----------------------------------------------------
>                 Key: HDFS-270
>                 URL: https://issues.apache.org/jira/browse/HDFS-270
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: 0.20.2
>            Reporter: Stu Hood
>            Assignee: Hairong Kuang
> I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a little slowly.
> The main reason the upgrade took so long was the block upgrades on the datanodes. Each
of our datanodes has 3 drives listed for the dfs.data.dir parameter. From looking at the logs,
it is fairly clear that the upgrade procedure does not attempt to upgrade all listed dfs.data.dir's
in parallel.
> I think even if all of your dfs.data.dir's are on the same physical device, there would
still be an advantage to performing the upgrade process in parallel. The less downtime, the
better: especially if it is potentially 20 minutes versus 60 minutes.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message