Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@gmail.com
 designates 209.85.215.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPzbcGSksN-CTNVWN_=7T6N8pu3BBu3VRtK7-vOF_4pdWzD-yw@mail.gmail.com>
References: 
 <CA+4kjVsPWQesEwBgJpQ3rALrSu5zpCeFtWt7fafuGiU210owYQ@mail.gmail.com>
	<CANpmDOfXa1sBezJkeXdqf8JBgAgiAFUrURF54-98v6b+98pURw@mail.gmail.com>
	<CA+4kjVvve63mg9KS+MFAGFn7oYyV1-ToWDwyzG9aOdUA__gH4g@mail.gmail.com>
	<CANpmDOfQA9eK+6me9Hzdk6JaYYthxU55dSNUo+o0a0_msFAeZA@mail.gmail.com>
	<CA+4kjVt4bmtfab_Ys9xjuTyZBLNnniPyh1r8ONBfTZrjDpcUXg@mail.gmail.com>
	<CAOyNEs5=u=H6xKPWC0ZUANGSMyNMLd__5zF9Actr4as7_vf4OQ@mail.gmail.com>
	<CANpmDOfyXb_KrX-WLuDxRQk3iybMhaiJ1i=1si4J4ZC4y=FXJw@mail.gmail.com>
	<CAPzbcGSksN-CTNVWN_=7T6N8pu3BBu3VRtK7-vOF_4pdWzD-yw@mail.gmail.com>
Date: Fri, 17 Jan 2014 11:10:45 +0100
Message-ID: 
 <CAFk14gv+YR1Wbc_9mwzQW96DUnujxZ076asrgmbSJSHgRJYYPg@mail.gmail.com>
Subject: Re: Hoya Proposal
From: Bernd Fondermann <bernd.fondermann@gmail.com>
To: "general@incubator.apache.org" <general@incubator.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Very informative explanation. I've got a better understanding of the
proposal now.

I suggest that a remark like this is added to the Hoya proposal, so this
part of the discussion becomes part of the vote on the proposal.The
proposal's "Known Risk" section lacks a "Relationships with Other Apache
Products" paragraph, which would be a good place to add this information.
There are more considerations missing that we have in other proposals. I'd
suggest to add them, too, to complete the proposal.

All-in-all I'm currently -1 on the proposal as it is, but ready to change
to +1 as this discussion reaches a consent.

(side note: When a project graduates, a PMC is established and tasked with
a specific task. Two PMCs cannot have the same task. Therefore, it is
better in my opinion to either clearly differentiate the projects from
start, or join them from start.)

Thanks,

  Bernd

On Fri, Jan 17, 2014 at 8:48 AM, Devaraj Das <ddas@hortonworks.com> wrote:
> Andreas, to me, Twill is a library, a convenience library, that one
> can use to write Yarn apps. Hoya aims to provide a general framework
> using which one can take existing apps (HBase/Accumulo to start with),
> and make them run well in a Yarn cluster, without intruding at all
> into the App internals. The goal is to have minimal, ideally zero
> changes, in the App itself. The only glue between the App and Hoya is
> a Hoya interface that the App needs to implement for it to be
> deployable/manageable by Hoya.
>
> Although, one could argue that Hoya can be written using Twill
> libraries (which is something we should pursue as part of
> long/medium-term collaboration between the two projects), but I'd
> argue that the goals of the two projects are different - Twill would
> continue to make Yarn app developers' lives easier, while Hoya is a
> tool that could deploy distributed-frameworks easily in a Yarn
> cluster, and be able to later do basic management (as is talked about
> in the github docs). Things like dynamic configuration patching for
> the applications like HBase to easily run in a Yarn cluster, security
> issues, failure handling and defining a model for reacting to
> failures, being able to store some state about applications to
> facilitate better application restart behavior in a Yarn cluster, etc.
> would be in the purview of Hoya. Management frameworks could use Hoya
> as a tool as well to talk to Yarn and do application
> start/stop/shrink/expand an instance of an application (e.g. HBase)
> cluster.
>
> On Thu, Jan 16, 2014 at 4:38 PM, Andreas Neumann <anew@apache.org> wrote:
>> I do see a lot of value in all the features of Hoya, and I am not trying to
>> discredit it. Yet I do think that most of these features are actually
>> already in Twill or would be great additions to Twill, and sooner or later
>> will be implemented in Twill, implying even greater overlap.
>>
>> I am not sure whether I follow the distinction between porting an existing
>> app and developing a new app, if both require similar abstractions... And
>> after all, porting an existing application is one way to start developing a
>> Yarn application.
>>
>> Anyway, I just wanted to point out the overlap and offer collaboration. If
>> the community thinks that the two projects are different beasts, that's
>> fine with me.
>>
>>
>> On Wed, Jan 15, 2014 at 10:40 AM, Josh Elser <josh.elser@gmail.com> wrote:
>>
>>> I wanted to weigh in on some of Steve's thoughts, I'm actually really
>>> excited about being able to leverage Hoya inside of Accumulo itself.
>>>
>>> We presently have a few system tests that rely on manual set up, which can
>>> be frustrating to deal with on a beefy boxes (running multiple Accumulo
>>> procs on a single host). Hoya drastically reduces the amount of effort it
>>> takes to run these distributed tests in a 'closer to real' environment.
>>> On Jan 15, 2014 8:36 AM, "Steve Loughran" <stevel@hortonworks.com> wrote:
>>>
>>> > On 15 January 2014 02:13, Andreas Neumann <anew@apache.org> wrote:
>>> >
>>> > > I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
>>> > > other type of existing application? If so, won't it have some common
>>> > > abstraction that is shared by all of them? That is where I see the
>>> > > similarity with Twill.
>>> > >
>>> > >
>>> > it started off as pure hbase, but now has the notion of a provider which
>>> > has a client-side and server side
>>> >
>>> > client side
>>> >  -helps set up the template JSON file to describe the cluster (e.g. adds
>>> > default values),
>>> >  -patches the configuration directory supplied at creation time with
>>> > whatever options it wants (e.g links up fs.default.name & ZK settings in
>>> > hbase-site.xml)
>>> >  -does preflight validation of cluster options
>>> >  -can also add its own .tar.gz to the resources of the AM (or any other
>>> > resources)
>>> >
>>> > server side
>>> >  -runs in the AM and sets up everything needed to run instances of a role
>>> > (e.g. HBase master, Accumulo GC, ..)
>>> >  -can run code in the AM to help set things up (e.g. Accumulo provider
>>> > service runs bin/accumulo init if needed)
>>> >  -TODO: liveness monitoring
>>> >
>>> > the requirements of an app to be deployable are
>>> >
>>> >
>>> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md
>>> >
>>> >
>>> >
>>> > > Whereas, if it is a separate effort for each existing application, say
>>> > > HBase, then what is the end goal for Hoya when it comes out of
>>> > incubation?
>>> > > To merge it back into HBase proper?
>>> > >
>>> > >
>>> > Now that it supports >1 application, it can't go into HBase. The
>>> individual
>>> > provider services can (their implementation classes are worked out via
>>> the
>>> > configuration.XML).
>>> >
>>> > But as we use the accumulo and HBase apps for testing, its really good to
>>> > have them both in the hoya project right now -a project that builds
>>> > downstream of both and needs
>>> > to be given the Hadoop filesystem paths to .tar.gz files of each app.
>>> >
>>> > There's nothing to stop 3rd party apps joining in, indeed, one thing I'd
>>> > like someone to do is actually have Hoya deploy SmartFrog .tar.gz files
>>> and
>>> > pass down configurations that deploy applications -for example
>>> jetty-based
>>> > web servers. That'd take an intern working at HP Labs over the summer,
>>> > maybe
>>> >
>>> > -steve
>>> >
>>> > --
>>> > CONFIDENTIALITY NOTICE
>>> > NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> > which it is addressed and may contain information that is confidential,
>>> > privileged and exempt from disclosure under applicable law. If the reader
>>> > of this message is not the intended recipient, you are hereby notified
>>> that
>>> > any printing, copying, dissemination, distribution, disclosure or
>>> > forwarding of this communication is strictly prohibited. If you have
>>> > received this communication in error, please contact the sender
>>> immediately
>>> > and delete it from your system. Thank You.
>>> >
>>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org