Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CADbBEnv8X=_vYeazCeR_U2fHntvNCkduGOPc9hT2FNzG=iEBDA@mail.gmail.com>
References: <140D71EF-F7D6-4D7C-B370-401A8561164E@hortonworks.com>
	<CALwhT956PunRg1i_OvKv64NCnNARGRQWwLZ8p5aGhRPrYU173w@mail.gmail.com>
	<CADbBEnv8X=_vYeazCeR_U2fHntvNCkduGOPc9hT2FNzG=iEBDA@mail.gmail.com>
Date: Mon, 30 May 2016 21:44:03 -0700
Message-ID: <CALwhT97twkNkcbUnk4JtmaZFVhh8m2ZwM6e_ZMWaZYciwTortQ@mail.gmail.com>
Subject: Re: [DISCUSS] what exactly are the stability guarantees of the YARN APIs
From: Karthik Kambatla <kasha@cloudera.com>
To: Sangjin Lee <sjlee0@gmail.com>
Cc: Steve Loughran <stevel@hortonworks.com>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a1149333cbd87a405341c0531
archived-at: Tue, 31 May 2016 04:44:09 -0000

--001a1149333cbd87a405341c0531
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Inline.

On Sat, May 28, 2016 at 11:34 AM, Sangjin Lee <sjlee0@gmail.com> wrote:

> I think there is more to it. The InterfaceStability javadoc states:
>     Incompatible changes must not be made to classes marked as stable.
>
> And in practice, I don't think the class annotation can be considered a
> simple sum of method annotations. There is a notion of class compatibilit=
y
> distinct from method stability. One key example is interfaces and abstrac=
t
> classes as in this case. The moment a new abstract method is added, the
> class becomes incompatible as it would break all downstream subclasses or
> implementing classes. That's the case even if *all methods are declared
> stable*. Thus, adding any abstract method (no matter what their
> scope/stability is) should be considered in violation of the stable
> contract of the class.
>

Fair point. I was referring to them in the context of adding @Evolving
methods to @Stable classes. Our policy states that "Classes not annotated
are implicitly =E2=80=9CPrivate=E2=80=9D. Class members not annotated inher=
it the
annotations of the enclosing class." So, the annotation on a method
overrides that of the enclosing class. This seems pretty reasonable to me.

Do you think there is reason to revisit this? If yes, we should update this
for Hadoop 3.


>
> Regarding interfaces and abstract classes, one future enhancement to the
> InterfaceStability annotation we could consider is formally separating th=
e
> contract for users of the API and the implementers of the API. They follo=
w
> different rules. It could be feasible to have an interface as Public/Stab=
le
> for users (anyone can use the API in a stable manner) but Private for
> implementers. The idea is that it is still a public interface but no
> third-party code should not subclass or implement it. I suspect a fair
> amount of hadoop's public interface might fall into that category. That
> itself is probably an incompatible change, so we might have to wait until
> after 3.0, however.
>

Interesting thought. Agree that we do not anticipate users sub-classing
most of our Public-Stable classes.

There are also classes which we do not anticipate end-users to directly
use, but devs might want to sub-class. This applies to pluggable entities;
e.g. SchedulingPolicy in fairscheduler. We are currently using
Public-Evolving to capture this intent.

Should we add a third annotation in addition to Audience and Stability to
capture whether a class can be extended? Given the few classes we
anticipate being extended, this is likely lesser work. :)


>
>
> On Sat, May 28, 2016 at 10:07 AM, Karthik Kambatla <kasha@cloudera.com>
> wrote:
>
>> Inline.
>>
>> On Sat, May 28, 2016 at 3:51 AM, Steve Loughran <stevel@hortonworks.com>
>> wrote:
>>
>> >
>> > In SLIDER-1130, I discovered that some changes to some YARN classes ha=
d
>> > broken my test code =E2=80=94new abstract methods had been defined, wh=
ich broke
>> the
>> > subclasses I have to mock a YARN cluster. These are invaluable classes=
,
>> as,
>> > once eventually the hadoop regressions had been worked around, they
>> showed
>> > that the label code was broken ( YARN-5135 ) and the fixYARN-4991 not
>> > backported to branch-2
>> >
>> > If the classes that had changed were tagged as @Private or @Evolving,
>> I'd
>> > have just said, "oh well, never mind"
>> >
>> > Except, the two interfaces that broke: ContainerStatus and NodeReport
>> =E2=80=94are
>> > marked @Public, @Stable
>> >
>> > Given that my code broke, I don't believe that the guarantees of a
>> @Stable
>> > interface, "Can evolve while retaining compatibility for minor release
>> > boundaries; in other words, incompatible changes to APIs marked Stable
>> are
>> > allowed only at major releases (i.e. at m.0)." are being kept.
>> >
>> > Now, in YARN-5130, we now have a patch to remark the interfaces as
>> > Unstable.
>> >
>> > However, while I can apply that patch myself, I want to raise a broade=
r
>> > question: what does the Yarn team consider constitutes "stable"?
>> >
>> >  And do that changes that took place in YARN-3886 and YARN-2882 meet
>> those
>> > requirements? New methods are going in, some of which are now being
>> tagged
>> > as @Evolving?
>>
>>
>> > I fail to see how something marked as @Stable can add @Evolving method=
s,
>> > and given that changes are breaking downstream code, I worry that not
>> > enough care is being taken over the maintenance of the guarantee which
>> > @Public, @Stable interfaces make to users
>> >
>>
>> Are you referring to adding @Evolving methods to @Stable classes? If yes=
,
>> my understanding is - the annotations are per-method and the class
>> annotations are a convenience to avoid having to annotate each method.
>>
>>
>> >
>> >
>> >
>> > -Steve
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> >
>> >
>>
>
>

--001a1149333cbd87a405341c0531--