hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6800) Detemine how Datanode layout changes should interact with rolling upgrade
Date Fri, 01 Aug 2014 00:37:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081743#comment-14081743

Colin Patrick McCabe commented on HDFS-6800:

Just to give a bit of additional context here: [~james.thomas] did some benchmarks that showed
that "creating 200k hard links (100k blocks) take just over a second on a dual core PC" when
using the optimized hard link code.  On a performance basis, I think it's very feasible to
support rolling upgrade to new DN layout versions.

If we don't choose to support this, we are going to make it very hard to evolve the DN code
in the future.  We would then require a major version change (i.e. Hadoop 3.0) to make any
major DN changes.  So I think we should just change the documentation a bit and support this
in the obvious way... by having the users call {{datanode \-rollback}} during a rolling rollback
if needed.  What do you guys think?

> Detemine how Datanode layout changes should interact with rolling upgrade
> -------------------------------------------------------------------------
>                 Key: HDFS-6800
>                 URL: https://issues.apache.org/jira/browse/HDFS-6800
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.6.0
>            Reporter: Colin Patrick McCabe
> We need to handle attempts to rolling-upgrade the DataNode to a new storage directory
> One approach is to disallow such upgrades.  If we choose this approach, we should make
sure that the system administrator gets a helpful error message and a clean failure when trying
to use rolling upgrade to a version that doesn't support it.  Based on the compatibility guarantees
described in HDFS-5535, this would mean that *any* future DataNode layout changes would require
a major version upgrade.
> Another approach would be to support rolling upgrade from an old DN storage layout to
a new layout.  This approach requires us to change our documentation to explain to users that
they should supply the {{\-rollback}} command on the command-line when re-starting the DataNodes
during rolling rollback.  Currently the documentation just says to restart the DataNode normally.
> Another issue here is that the DataNode's usage message describes rollback options that
no longer exist.  The help text says that the DN supports {{\-rollingupgrade rollback}}, but
this option was removed by HDFS-6005.

This message was sent by Atlassian JIRA

View raw message