Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: [DISCUSS] Treating LimitedPrivate({"MapReduce"}) as Public APIs for YARN applications
From: Hitesh Shah <hitesh@apache.org>
In-Reply-To: <89004A0B-7B3F-4EF9-B944-4F17E7182C57@yahoo.com>
Date: Tue, 10 May 2016 11:34:27 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <617F466E-4877-4DE2-8467-EE5C1FA921F5@apache.org>
References: <650A799F-E26D-401F-91CE-9A64472B6532@apache.org> <89004A0B-7B3F-4EF9-B944-4F17E7182C57@yahoo.com>
To: yarn-dev@hadoop.apache.org,
 mapreduce-dev@hadoop.apache.org,
 common-dev@hadoop.apache.org
archived-at: Tue, 10 May 2016 18:34:31 -0000

There seems to be some incorrect assumptions on why the application had =
an issue. For rolling upgrade deployments, the application bundles the =
client-side jars that it was compiled against and uses them in its =
classpath and expects to be able to communicate with upgraded servers. =
Given that hadoop-common is a monolithic jar, it ends up being used on =
both client-side and server-side. The problem in this case was caused by =
the fact that the ResourceManager was generating the credentials file =
with a format understood only by hadoop-common from 3.x. For an =
application compiled against 2.x and has *only* hadoop-common from 2.x =
on its classpath, trying to read this file fails. =20

This is not about whether internal implementations can change for =
non-public APIs. The file format for the Credential file in this =
scenario is *not* internal implementation especially when you can have =
different versions of the library trying to read the file. If an older =
client is talking to a newer versioned server, the general backward =
compat assumption is that the client should receive a response that it =
can parse and understand. In this scenario, the credentials file =
provided to the YARN app by the RM should have been written out with the =
older version or at the very least been readable by the older =
hadoop-common.jar.

In any case, does anyone have any specific concerns with changing =
LimitedPrivate({"MapReduce"}) to Public?

And sure, if we are saying that Hadoop-3.x requires all apps built =
against it to go through a full re-compile as well as downtime as =
existing apps may no longer work out of the box, lets call it out very =
explicitly in the Release notes.=20

=E2=80=94 Hitesh

> On May 10, 2016, at 9:24 AM, Allen Wittenauer =
<allenwittenauer@yahoo.com> wrote:
>=20
>=20
>> On May 10, 2016, at 8:37 AM, Hitesh Shah <hitesh@apache.org> wrote:
>>=20
>> There have been various discussions on various JIRAs where upstream =
projects such as YARN apps ( Tez, Slider, etc ) are called out for using =
the above so-called Private APIs. A lot of YARN applications that have =
been built out have picked up various bits and pieces of implementation =
from MapReduce and DistributedShell to get things to work.
>>=20
>> A recent example is a backward incompatible change introduced ( where =
the API is not even directly invoked ) in the Credentials class related =
to the ability to read tokens/credentials from a file.
>=20
> 	Let=E2=80=99s be careful here.  It should be noted that the =
problem happened primarily because the application jar appears to have =
included some hadoop jars in them.   So the API invocation isn=E2=80=99t =
the problem:  it=E2=80=99s the fact that the implementation under the =
hood changed.  If the application jar didn=E2=80=99t bundle hadoop jars =
=E2=80=94especially given that were already on the classpath--this =
problem should never have happened.
>=20
>> This functionality is required by pretty much everyone as YARN =
provides the credentials to the app by writing the credentials/tokens to =
a local file which is read in when UserGroupInformation.getCurrentUser() =
is invoked.
>=20
> 	What you=E2=80=99re effectively arguing is that implementations =
should never change for public (and in this case LimitedPrivate) APIs.  =
I don=E2=80=99t think that=E2=80=99s reasonable.  Hadoop is filled with =
changes in major branches where the implementations have changed but the =
internals have been reworked to perform the work in a slightly different =
manner.
>=20
>> This change breaks rolling upgrades for yarn applications from 2.x to =
3.x (whether we end up supporting rolling upgrades across 2.x to 3.x is =
a separate discussion )
>=20
> =09
> 	At least today, according to the document attached to YARN-666 =
(lol), rolling upgrades are only supported within the same major =
version. =20
>=20
>>=20
>> I would like to change our compatibility docs to state that any API =
that is marked as LimitedPrivate{Mapreduce} implies LimitedPrivate{YARN =
Applications}.
>>=20
>> Comments/concerns?=20
>=20
>=20
> 	a)  That isn=E2=80=99t good enough.  No one reads the =
compatibility guidelines as it is given how many times someone says =
=E2=80=9CX=E2=80=9D isn=E2=80=99t covered when it clearly is.
>=20
> 	b) LimitedPrivate{=E2=80=9CYARN Applications=E2=80=9D} makes =
zero sense.  At that point it=E2=80=99s Public and the source should be =
changed to reflect that.  Especially given those flags impacts things =
like how the javadocs are generated.


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org