hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: Towards Hadoop 1.0: Stronger API Compatibility from 0.21 onwards
Date Mon, 28 Sep 2009 19:04:29 GMT
I think we should not require Job Q compatibility for 1.0 release.

thanks,
dhruba


On Mon, Sep 28, 2009 at 11:06 AM, Sanjay Radia <sradia@yahoo-inc.com> wrote:

>
> On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote:
>
>  Dhruba Borthakur wrote:
>> > It is really nice to have wire-compatibility between clients and servers
>> > running different versions of hadoop. The reason we would like this is
>> > because we can allow the same client (Hive, etc) submit jobs to two
>> > different clusters running different versions of hadoop. But I am not
>> stuck
>> > up on the name of the release that supports wire-compatibility, it can
>> be
>> > either 1.0  or something later than that.
>> > API compatibility  +1
>> > Data compatibility +1
>> > Job Q compatibility -1Wire compatibility +0
>>
>>
>> That's stability of the job submission network protocol you are looking
>> for there.
>>  * We need a job submission API that is designed to work over long-haul
>> links and versions
>>  * It does not have to be the same as anything used in-cluster
>>  * It does not actually need to run in the JobTracker. An independent
>> service bridging the stable long-haul API to an unstable datacentre
>> protocol does work, though authentication and user-rights are a
>> troublespot
>>
>>
>
>
> I think you are misinterpreting what Job Q compatibility means.
> It is about jobs already in the queue surviving an upgrade across a
> release.
>
> See my initial proposal on Jan 16th:
>
> https://issues.apache.org/jira/browse/HADOOP-5071?focusedCommentId=12664691&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
> #action_12664691
>
> Doug argued that it is nice to have but not required for 1.0 - can be added
> later.
>
>
> sanjay
>
>
>> Similarly, it would be good for a stable long-haul HDFS protocol, such
>> as FTP or webdav. Again, no need to build into the namenode .
>>
>> see http://www.slideshare.net/steve_l/long-haul-hadoop
>> and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop
>>
>>
>


-- 
Connect to me at http://www.facebook.com/dhruba

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message