hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: 0.20.2 HDFS incompatible with 0.20.1
Date Tue, 05 Jan 2010 20:06:19 GMT
On 1/5/10 11:29 AM, "Todd Lipcon" <todd@cloudera.com> wrote:
> 1) Although we certainly do not guarantee wire compatibility between minor
> versions (0.20 -> 0.21) have we previously implied wire compatibility
> between bugfix releases?

IIRC, it has been implied and was a goal but not officially written anywhere
public that I know of.

> 2) Is the above something we *should* be guaranteeing already?

A) From an ops perspective, the lack of compatibility between even minor
releases is a pain.

B) Most folks with even slightly complex environments are likely patching
Hadoop.  A good chunk of those are likely in ways that breaks compatibility.
[For example, we're working on a TCP buffer patch for HDFS to fix what we
suspect is a latency problem.  Does it break compat?  Maybe.]

> 3) If we haven't guaranteed the above, how many users think we have? (ie how
> do we correctly call out this fact in the 0.20.2 release notes in such a way
> that no one gets surprised).

I suspect most folks don't even know that micros are incompatible until they
suddenly realize that distcp doesn't work.

> I can imagine plenty of organizations where a
> lockstep upgrade between client and server is difficult, and we should make
> sure that cluster operators know it will be necessary. Since it wasn't
> necessary between 0.20.0 and 0.20.1, or various 0.18 releases, people may
> have grown used to non-lockstep upgrades.

I can easily see many organizations only upgrading when it breaks due to the
Hadoop binaries being spread far and wide  and under the control of many
different departments without any sort of centralized management.

Despite the development model, I doubt few ever do just a mapred or hdfs
upgrade, so any change in the stack will likely trigger a full Hadoop

> 4) If the above are problems, would it be worth considering a patch for
> branch-20 that provides a client that is compatible with either, based on
> the datanode protocol version number of the server? It seems like a bit of
> scary complexity, but wanted to throw it out there.

Everyone knows I don't mind playing devil's advocate :), so let me ask the
obvious question:

Bugs are bad, etc, etc, but is it so critical that it has to be in the 0.20
branch at all?

I'd rather see the community spend cycles on 0.21 than worrying about 0.20
given that we're fast approaching the 1yr birthday of 0.20.0....

View raw message