hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
Date Tue, 10 Jan 2017 22:40:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816405#comment-15816405

Sean Mackrory commented on HDFS-11096:

After looking at where SortedMapWritable is used some more, I'm more convinced it's only a
concern if we care about source compatibility, which is not required for rolling upgrades.
I also had a second look at wire compatibility and I found a few concerning things I'll look
at and possibly fix:

* The message getHdfsBlockLocations has disappeared as well as related types
* The field nonDfsUsed in DatanodeInfoProto changed from index 9 to index 15

For YARN (CC [~kasha]), the field nodeLabels in several structures in yarn_protos changed
from string to a custom type, and memory in ResourceProto change from int32 to int64 (not
sure if it's the case in protobuf, but that may not be incompatible?)

There's also a lot of messages moving between files, but not otherwise changing in any incompatible
way. That's not a concern is it?

If anyone else wants to see the changes in protobuf, this is what I did (if anything, you'll
want to replace meld with your own diff-tool-of-choice):

#!/usr/bin/env bash

cd /tmp


mkdir new
mkdir old
git clone git://git.apache.org/hadoop.git

function gather_protos() {
  for proto in $(cd ${SOURCE} && find . -name \*.proto | sed -e 's|^\./||'); do
    #flattened=${proto//\//_} # Trips up on files that moved
    flattened=$(basename ${proto})
    cp ${SOURCE}/${proto} ${TARGET}/${flattened}

(cd hadoop; git checkout ${OLD})
gather_protos hadoop old

(cd hadoop; git checkout ${NEW})
gather_protos hadoop new

meld old new

> Support rolling upgrade between 2.x and 3.x
> -------------------------------------------
>                 Key: HDFS-11096
>                 URL: https://issues.apache.org/jira/browse/HDFS-11096
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: rolling upgrades
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
>            Priority: Blocker
> trunk has a minimum software version of 3.0.0-alpha1. This means we can't rolling upgrade
between branch-2 and trunk.
> This is a showstopper for large deployments. Unless there are very compelling reasons
to break compatibility, let's restore the ability to rolling upgrade to 3.x releases.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message