hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-270) DFS Upgrade should process dfs.data.dirs in parallel
Date Thu, 17 Jul 2014 18:51:06 GMT

     [ https://issues.apache.org/jira/browse/HDFS-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth resolved HDFS-270.

    Resolution: Won't Fix

I agree with resolving this.  I'm resolving this as won't fix, but others can feel free to
reopen if anyone thinks something is still missing.

If we look at rolling upgrades, probably the closest analogous thing is finalization of the
upgrade, when we delete the "trash" block files that users deleted before the upgrade was
finalized.  I just reviewed the code for this, and we do the delete in a separate thread per

> DFS Upgrade should process dfs.data.dirs in parallel
> ----------------------------------------------------
>                 Key: HDFS-270
>                 URL: https://issues.apache.org/jira/browse/HDFS-270
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: 0.20.2
>            Reporter: Stu Hood
>            Assignee: Hairong Kuang
> I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a little slowly.
> The main reason the upgrade took so long was the block upgrades on the datanodes. Each
of our datanodes has 3 drives listed for the dfs.data.dir parameter. From looking at the logs,
it is fairly clear that the upgrade procedure does not attempt to upgrade all listed dfs.data.dir's
in parallel.
> I think even if all of your dfs.data.dir's are on the same physical device, there would
still be an advantage to performing the upgrade process in parallel. The less downtime, the
better: especially if it is potentially 20 minutes versus 60 minutes.

This message was sent by Atlassian JIRA

View raw message