incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devaraj Das <d...@hortonworks.com>
Subject Re: Hoya Proposal
Date Fri, 17 Jan 2014 07:48:15 GMT
Andreas, to me, Twill is a library, a convenience library, that one
can use to write Yarn apps. Hoya aims to provide a general framework
using which one can take existing apps (HBase/Accumulo to start with),
and make them run well in a Yarn cluster, without intruding at all
into the App internals. The goal is to have minimal, ideally zero
changes, in the App itself. The only glue between the App and Hoya is
a Hoya interface that the App needs to implement for it to be
deployable/manageable by Hoya.

Although, one could argue that Hoya can be written using Twill
libraries (which is something we should pursue as part of
long/medium-term collaboration between the two projects), but I'd
argue that the goals of the two projects are different - Twill would
continue to make Yarn app developers' lives easier, while Hoya is a
tool that could deploy distributed-frameworks easily in a Yarn
cluster, and be able to later do basic management (as is talked about
in the github docs). Things like dynamic configuration patching for
the applications like HBase to easily run in a Yarn cluster, security
issues, failure handling and defining a model for reacting to
failures, being able to store some state about applications to
facilitate better application restart behavior in a Yarn cluster, etc.
would be in the purview of Hoya. Management frameworks could use Hoya
as a tool as well to talk to Yarn and do application
start/stop/shrink/expand an instance of an application (e.g. HBase)
cluster.

On Thu, Jan 16, 2014 at 4:38 PM, Andreas Neumann <anew@apache.org> wrote:
> I do see a lot of value in all the features of Hoya, and I am not trying to
> discredit it. Yet I do think that most of these features are actually
> already in Twill or would be great additions to Twill, and sooner or later
> will be implemented in Twill, implying even greater overlap.
>
> I am not sure whether I follow the distinction between porting an existing
> app and developing a new app, if both require similar abstractions... And
> after all, porting an existing application is one way to start developing a
> Yarn application.
>
> Anyway, I just wanted to point out the overlap and offer collaboration. If
> the community thinks that the two projects are different beasts, that's
> fine with me.
>
>
> On Wed, Jan 15, 2014 at 10:40 AM, Josh Elser <josh.elser@gmail.com> wrote:
>
>> I wanted to weigh in on some of Steve's thoughts, I'm actually really
>> excited about being able to leverage Hoya inside of Accumulo itself.
>>
>> We presently have a few system tests that rely on manual set up, which can
>> be frustrating to deal with on a beefy boxes (running multiple Accumulo
>> procs on a single host). Hoya drastically reduces the amount of effort it
>> takes to run these distributed tests in a 'closer to real' environment.
>> On Jan 15, 2014 8:36 AM, "Steve Loughran" <stevel@hortonworks.com> wrote:
>>
>> > On 15 January 2014 02:13, Andreas Neumann <anew@apache.org> wrote:
>> >
>> > > I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
>> > > other type of existing application? If so, won't it have some common
>> > > abstraction that is shared by all of them? That is where I see the
>> > > similarity with Twill.
>> > >
>> > >
>> > it started off as pure hbase, but now has the notion of a provider which
>> > has a client-side and server side
>> >
>> > client side
>> >  -helps set up the template JSON file to describe the cluster (e.g. adds
>> > default values),
>> >  -patches the configuration directory supplied at creation time with
>> > whatever options it wants (e.g links up fs.default.name & ZK settings in
>> > hbase-site.xml)
>> >  -does preflight validation of cluster options
>> >  -can also add its own .tar.gz to the resources of the AM (or any other
>> > resources)
>> >
>> > server side
>> >  -runs in the AM and sets up everything needed to run instances of a role
>> > (e.g. HBase master, Accumulo GC, ..)
>> >  -can run code in the AM to help set things up (e.g. Accumulo provider
>> > service runs bin/accumulo init if needed)
>> >  -TODO: liveness monitoring
>> >
>> > the requirements of an app to be deployable are
>> >
>> >
>> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md
>> >
>> >
>> >
>> > > Whereas, if it is a separate effort for each existing application, say
>> > > HBase, then what is the end goal for Hoya when it comes out of
>> > incubation?
>> > > To merge it back into HBase proper?
>> > >
>> > >
>> > Now that it supports >1 application, it can't go into HBase. The
>> individual
>> > provider services can (their implementation classes are worked out via
>> the
>> > configuration.XML).
>> >
>> > But as we use the accumulo and HBase apps for testing, its really good to
>> > have them both in the hoya project right now -a project that builds
>> > downstream of both and needs
>> > to be given the Hadoop filesystem paths to .tar.gz files of each app.
>> >
>> > There's nothing to stop 3rd party apps joining in, indeed, one thing I'd
>> > like someone to do is actually have Hoya deploy SmartFrog .tar.gz files
>> and
>> > pass down configurations that deploy applications -for example
>> jetty-based
>> > web servers. That'd take an intern working at HP Labs over the summer,
>> > maybe
>> >
>> > -steve
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message