incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Hoya Proposal
Date Fri, 17 Jan 2014 09:34:11 GMT
> Andreas, to me, Twill is a library, a convenience library, that one
> can use to write Yarn apps. Hoya aims to provide a general framework
> using which one can take existing apps (HBase/Accumulo to start with),
> and make them run well in a Yarn cluster, without intruding at all
> into the App internals. The goal is to have minimal, ideally zero
> changes, in the App itself. The only glue between the App and Hoya is
> a Hoya interface that the App needs to implement for it to be
> deployable/manageable by Hoya.
>
> Although, one could argue that Hoya can be written using Twill
> libraries (which is something we should pursue as part of
> long/medium-term collaboration between the two projects),


FWIW the first iteration was 100% groovy, using my 2012 "Grumpy" code as a
starting point https://github.com/steveloughran/grumpy -

The later iterations are all Java.

The current design tries to have a clean model-view split server side, with
all the server state stored independently from the AM itself,

https://github.com/hortonworks/hoya/tree/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/state

This state is something that could perhaps be worked out into something
more re-usable, as all it really does is

-take a specification of role types, the no. of instances of each, and
some options (there's an unused anti-affinity flag)
- build a model of the cluster, including live containers, queued
operations, nodes in the YARN cluster and their history
- Build up a list of actions: requests and releases aimed at keeping
the model (and hence the deployed app) consistent with the
specification.
- persist placement history to HDFS for best-effort reassignment
requests on cluster restart ( see
https://github.com/hortonworks/hoya/blob/master/src/site/markdown/rolehistory.md
)
- Respond to relayed events (container assignment, failure, rebuild
after AM restart by updating model

For any long-lived YARN service, the goal "keep the number of
instances of a role constant" is an underlying use case, from
Distributed Shell up. Hoya just effectively allows this to be driven
by an JSON document persisted to HDFS, updated via RPC, and delegates
to plugin providers the work of actually starting processes at the far
end.

I've love to see this pulled out and re-usable, indeed, I'd love to
see the YARN dshell architecture redone a bit more cleanly too:
https://issues.apache.org/jira/browse/YARN-1360

Not only would that give something for others to use, but there are
some aspect of failure handling that we can/should improve in a way
that would benefit all apps, such as better weighted moving average
failure tracking, sophisticated policies on reacting to it, and moving
beyond a simple works/doesn't work to a more nuanced greylist of
perceived server reliability:
http://steveloughran.blogspot.co.uk/2014/01/greylisting-like-blacklisting-only-more.html

If we can work with the Twill team to factor this stuff into something
broadly re-usable, that would be great for everyone.

-steve

ps. I'd a standard management service API too

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message