hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Thomas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade
Date Fri, 22 Aug 2014 18:54:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107286#comment-14107286

James Thomas commented on HDFS-6800:

[~arpitagarwal], thanks for the response. The reason we use the flag in all cases is that
we now support rolling upgrade rollbacks to an old DataNode layout version (a process that
in the non-rolling upgrade case requires this flag), so I thought it would be easiest to just
have administrators use the flag in all rollback cases (as you can see in the patch, there
is no performance overhead if the rollback is not to a previous DataNode layout version).
It's really just a convention, but I think it is simple and makes sense.

As for the testing, as [~cmccabe] mentioned in his first comment above, I have already run
tests with rolling upgrade to a new DN layout version with 100k blocks, and the hard link
time was very fast. I don't think restoring the previous directory during a rollback is very
expensive -- the only possible problem I see is the deletion of the current dir. Is that what
you were concerned about? If so, what is the budget for rollback time? I can't imagine the
deletion would take more than a couple tens of seconds.

> Determine how Datanode layout changes should interact with rolling upgrade
> --------------------------------------------------------------------------
>                 Key: HDFS-6800
>                 URL: https://issues.apache.org/jira/browse/HDFS-6800
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.6.0
>            Reporter: Colin Patrick McCabe
>            Assignee: James Thomas
>         Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch, HDFS-6800.patch
> We need to handle attempts to rolling-upgrade the DataNode to a new storage directory
> One approach is to disallow such upgrades.  If we choose this approach, we should make
sure that the system administrator gets a helpful error message and a clean failure when trying
to use rolling upgrade to a version that doesn't support it.  Based on the compatibility guarantees
described in HDFS-5535, this would mean that *any* future DataNode layout changes would require
a major version upgrade.
> Another approach would be to support rolling upgrade from an old DN storage layout to
a new layout.  This approach requires us to change our documentation to explain to users that
they should supply the {{\-rollback}} command on the command-line when re-starting the DataNodes
during rolling rollback.  Currently the documentation just says to restart the DataNode normally.
> Another issue here is that the DataNode's usage message describes rollback options that
no longer exist.  The help text says that the DN supports {{\-rollingupgrade rollback}}, but
this option was removed by HDFS-6005.

This message was sent by Atlassian JIRA

View raw message