hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards
Date Fri, 25 Sep 2009 17:13:14 GMT
It is really nice to have wire-compatibility between clients and servers
running different versions of hadoop. The reason we would like this is
because we can allow the same client (Hive, etc) submit jobs to two
different clusters running different versions of hadoop. But I am not stuck
up on the name of the release that supports wire-compatibility, it can be
either 1.0  or something later than that.
API compatibility  +1
Data compatibility +1
Job Q compatibility -1Wire compatibility +0


On Fri, Sep 25, 2009 at 10:05 AM, Doug Cutting <cutting@apache.org> wrote:

> Sanjay Radia wrote:
>> Both Facebook (Dhruba tells me) and Yahoo are suffering badly from the
>> lack of wire compatibility -  a major motivaiton
>> for Yahoo to develop Avro.
> Indeed.  Wire compatibility is a crucial feature that we should release as
> soon as possible.  Perhaps before 1.0 if 1.0 slips, perhaps after if we
> discover that it's harder to implement than we anticipate.
>  Wire compatibility - open question; but my thoughts are:
>>     With the progress we have made on Avro so far I think there is a very
>> good chance to get wire compatibility in 22 which we
>> can then call 0.99 or 1.0. I think it is worth a shot.
> +1 It's certainly worth a shot.
> 1.0 is fundamentally about being able to upgrade a cluster without changing
> application code, i.e., API compatibility.  Wire compatibility will let
> folks, e.g., use a single client library version to talk to clusters running
> different versions, a wonderful feature, but distinct from the fundamental
> goal of 1.0.
> In general we should not tie too many features to specific releases in
> advance of their implementation, since that causes releases to slip when
> features slip.  Rather, we should work hard to implement high-priority
> features and release periodically, as features are completed and we are able
> to qualify releases.  Long-term API compatibility is a very high-priority
> feature.  The first release that has APIs that we think we can support
> back-compatibly for perhaps a few years should be called 1.0.  Hopefully
> that will also have some other high-priority features, like security, wire
> compatibility, etc.  But I don't see the purpose of requiring a specific
> list of high-priority features besides API compatibility before we declare
> 1.0, and doing so could needlessly keep valuable features from users.
> Doug

Connect to me at http://www.facebook.com/dhruba

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message