hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
Date Fri, 15 Sep 2017 23:44:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168686#comment-16168686

Allen Wittenauer commented on HDFS-11096:

bq. how do you feel about set -v? 

It's usually overkill.  It typically means that the code isn't giving enough hints about what
it's doing or that it lacks a --debug option to tell me about important events if something
is going wrong and I need in depth help.

bq. And did you mean set +e?

No, I meant setting 'set -e' as was in the code.  The default for bash is set +e.

bq.  I feel like if it's okay for a command to sometimes fail, you can deal with that return
code explicitly, otherwise I'd like that failure to bubble up. Am I missing something?

set -e will exit bash with no chance to deal with the failure. That's disastrous if the code
actually knows how to exit out gracefully or has workarounds or multiple ways to try something
or wants to use boolean result codes.

bq. Not sure I understand exactly what hadoop_actual_ssh was supposed to be doing before,
but it's not used elsewhere and is marked as private

hadoop_actual_ssh is called from hadoop_connect_to_hosts_without_pdsh.  That code was originally
written with xargs, but it wasn't in a POSIX compatible way so the xargs implementation got
removed.  But when it was written with xargs, the problem was one can't call a function as
a parameter.  So ... other ways were invented.  You can see the original xargs implementation
in hadoop-user-functions.sh.example to see how I worked around it.

That said... I have a hunch that your changes probably break hadoop/hdfs/... --workers flag
either completely or just the output when pdsh isn't installed. 

An easier entry point is likely to be hadoop_connect_to_hosts which can take advantage of
pdsh and do things in parallel. Just need to set the appropriate HADOOP_WORKERS var.

bq. $(dirname $ {0}) 

That's shellcheck hinting that something may be wrong... ;)

I generally go for something like:

BINDIR=$(cd -P -- "$(dirname -- "${this}")" >/dev/null && pwd -P)

This fixes some potential issues:
* deals with most of the ridiculous aliasing that people tend to with cd
* leading dashes for directory names aren't a problem
* gives an absolute, symlink resolved path 
* .... and one of my absolute favorite obscure bugs: bash foo.sh now works if foo.sh is in
the path

$ cat ~/bin/foo.sh
echo "0: $0"
echo "BS0: ${BASH_SOURCE-$0}"
$ foo.sh
0: /Users/aw/bin/foo.sh
BS0: /Users/aw/bin/foo.sh
$ bash foo.sh
0: foo.sh
BS0: /Users/aw/bin/foo.sh

bq. Switch to using create-release - --native wasn't working because the Docker image doesn't
have a high enough version of cmake

Rebase required? Or is this some other Dockerfile? Because otherwise precommit would be failing...

> Support rolling upgrade between 2.x and 3.x
> -------------------------------------------
>                 Key: HDFS-11096
>                 URL: https://issues.apache.org/jira/browse/HDFS-11096
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: rolling upgrades
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
>            Assignee: Sean Mackrory
>            Priority: Blocker
>         Attachments: HDFS-11096.001.patch, HDFS-11096.002.patch, HDFS-11096.003.patch
> trunk has a minimum software version of 3.0.0-alpha1. This means we can't rolling upgrade
between branch-2 and trunk.
> This is a showstopper for large deployments. Unless there are very compelling reasons
to break compatibility, let's restore the ability to rolling upgrade to 3.x releases.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message