hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@effectivemachines.com>
Subject YARN native services Re: 2017-10-06 Hadoop 3 release status update
Date Tue, 10 Oct 2017 00:51:05 GMT

> On Oct 6, 2017, at 5:51 PM, Eric Yang <eyang@hortonworks.com> wrote:
> yarn application -deploy –f spec.json
> yarn application -stop <service-name>
> yarn application -restart <service-name>
> yarn application -remove <service-name>
> and
> yarn application –list will display both application list from RM as well as docker
	IMO, that makes much more sense. [*] I’m trying think of a reason why I’d care if something
was using this API or not.  It’s not like users can’t run whatever they want as part of
their job now.  The break out is really only necessary so I have an idea if something is running
that is using the REST API daemon. But more on that later….

> I think the development team was concerned that command structure overload between batch
applications and long running services.  In my view, there is no difference, they are all
applications.  The only distinction is the launching and shutdown of services may be different
from batch jobs.  I think user can get used to these command structures without creating additional
command grouping.

	I pretty much agree.  In fact, I’d love to see ‘yarn application’ even replace ‘yarn
jar’. One Interface To Rule Them All.

	I was under the impression (and, maybe this was my misunderstanding. if so, sorry) that “the
goal” for this first pass was to integrate the existing Apache Slider functionality into
YARN.  As it stands, I don’t think those goals have been met.  It doesn’t seem to be much
different than just writing a shell profile to call slider directly:

function yarn_subcommand_service
   exec slider “$@“

(or whatever). Plus doing it this way, one gets the added benefit of the SIGNIFICANTLY better
documentation. (Seriously: well done that team)

	From an outside perspective, the extra daemon for running the REST API seems like when it
should have clicked that the project is going off the rails and missing the whole “integration”
aspect. Integrating the REST API into the RM from day one and the command separation would
have also stuck out. If the RM runs the REST API, it now becomes a problem of “how does
a user launch more than just a jar easily?” A problem that Hadoop has had since nearly day
one.  Redefining the “application” subcommand sounds like a reasonable way to move forward
on that problem while also dropping the generic sounding "service" subcommand. 

	But all that said, it feels like direct integration was avoided from the beginning and I’m
unclear as to why. Take this line from the quick start documentation: 

	"Start all the hadoop components HDFS, YARN as usual.”

		a) This sentence is pretty much a declaration that this feature set isn’t part of “YARN”.

		b) Minimally, this should link to ClusterSetup. 

	Anyway, yes, please work on removing all of these extra adoption barriers and increased workload
on admin teams with Yet Another Daemon to monitor and collect metrics. 


[*] - I’m reminded of a conversation I had with a PMC member year or three ago about HDFS.
They proudly almost defiantly stated that the HDFS command structure is such because it resembles
the protocols and that was great. Guess what: users’ don’t care about how something is
implemented, much less the protocols that are used to drive it. They care about consistency,
EOU, and all those feel good things that make applications a joy to use. They have more important
stuff to do. Copying the protocols onto the command line only help the person who wrote it
and no one else. It’s hard not to walk away from playing with YARN in this branch as exhibiting
those same anti-user behaviors.

To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

View raw message