hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards
Date Fri, 25 Sep 2009 17:05:00 GMT
Sanjay Radia wrote:
> Both Facebook (Dhruba tells me) and Yahoo are suffering badly from the 
> lack of wire compatibility -  a major motivaiton
> for Yahoo to develop Avro.

Indeed.  Wire compatibility is a crucial feature that we should release 
as soon as possible.  Perhaps before 1.0 if 1.0 slips, perhaps after if 
we discover that it's harder to implement than we anticipate.

> Wire compatibility - open question; but my thoughts are:
>      With the progress we have made on Avro so far I think there is a 
> very good chance to get wire compatibility in 22 which we
> can then call 0.99 or 1.0. I think it is worth a shot.

+1 It's certainly worth a shot.

1.0 is fundamentally about being able to upgrade a cluster without 
changing application code, i.e., API compatibility.  Wire compatibility 
will let folks, e.g., use a single client library version to talk to 
clusters running different versions, a wonderful feature, but distinct 
from the fundamental goal of 1.0.

In general we should not tie too many features to specific releases in 
advance of their implementation, since that causes releases to slip when 
features slip.  Rather, we should work hard to implement high-priority 
features and release periodically, as features are completed and we are 
able to qualify releases.  Long-term API compatibility is a very 
high-priority feature.  The first release that has APIs that we think we 
can support back-compatibly for perhaps a few years should be called 
1.0.  Hopefully that will also have some other high-priority features, 
like security, wire compatibility, etc.  But I don't see the purpose of 
requiring a specific list of high-priority features besides API 
compatibility before we declare 1.0, and doing so could needlessly keep 
valuable features from users.


View raw message