hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: Locking protocols for hadoop-2.x.x
Date Fri, 20 Apr 2012 07:15:31 GMT
On Thu, Apr 19, 2012 at 11:46 PM, Arun C Murthy <acm@hortonworks.com> wrote:
> Moving to a separate thread...
>
> On Apr 20, 2012, at 1:24 AM, Todd Lipcon wrote:
>
>> On Thu, Apr 19, 2012 at 12:26 PM, Eli Collins <eli@cloudera.com> wrote:
>>
>>> On Thu, Apr 19, 2012 at 11:45 AM, Arun C Murthy <acm@hortonworks.com>
>>> wrote:
>>>> However, we should consider whether HDFS protocols are 'ready' for us to
>>> commit to them for the foreseeable future, my sense is that it's a tad
>>> early - particularly with auto-failover not complete.
>>>
>>> Agree that we're a little too early on the HDFS protocol side, think
>>> MR2 is probably in a similar boat wrt stability as well.
>>>
>
> Agreed, I didn't mean to point fingers at HDFS - it was just the most recent changes.
>
>> Regarding protocols:
>> +1 to _not_ locking down "cluster-internal" wire compatibility at this
>> point. i.e we can break DN<->NN, or NN<->SBN, or Admin command ->
NN
>> compatibility still.
>> +1 to locking down client wire compatibility with the release of 2.0. After
>> 2.0 is released I would like to see all 2.0.x clients continue to be
>> compatible. Now that we are protobuf-ified, I think this is doable.
>> Should we open a separate discussion thread for the above?
>
> Good points on separating client & internal protocols.
>
> My sense is that locking client-protocols is a great start, but not sufficient.
>
> Ideally, we should be considering things like rolling upgrades etc. which necessitate
compatibility all across. I'm fully aware it might be too early for us to lock them...
>

Yup, we've put the mechanism into HDFS for rolling upgrades
(HDFS-2983) and filed (MR-4150) for the same in MR2, but they'll only
be useful if we lock down the protocol (and use PB to get around
differences).  Agree w Todd that we're too early for those right now,
and they're much less painful breakages than client <-> server.

> Maybe we can do some hadoop-2.x-(alpha,beta) releases for a few months and then just
bite the bullet as HA & YARN protocols stabilize?
>

Sounds good, we should probably use eg "alpha1", "alpha2" etc in case
we need to do more than a single alpha or beta release.

Thanks,
Eli

> Thoughts?
>
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

Mime
View raw message