Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of josh.elser@gmail.com
 designates 209.85.214.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAL5zq9YdBo7bWxgn9_BGF+dQD3dbmDK-1LbRVqTg5zr4XFK1aQ@mail.gmail.com>
References: 
 <CAGHyZ6LCofhLaChRgifEJ9XxajVQLxbgnrOTvSmZf_-WqvcPNQ@mail.gmail.com>
	<CAGUtCHpyMQBguxfVJ2_So4qtao5wGQ8YX23RL3U4fMVWKsty2g@mail.gmail.com>
	<5332FC76.3000604@gmail.com>
	<CAGUtCHqF77Uh2w8zVyVSzaOYvYGY+z9oTY7u7wsofBRS5HV3Zw@mail.gmail.com>
	<5332FFD0.5030304@gmail.com>
	<CAGUtCHpTMLBp-Vhy9drmfPNZWyfvcQp4-ksqyUhXiv5OoKyi+A@mail.gmail.com>
	<53330473.7000700@gmail.com>
	<CAGUtCHppnMUsaDAMkLLFaor3Gy4TuSDYRDmprZv=X6L+cG9aBQ@mail.gmail.com>
	<53331700.6080407@gmail.com>
	<CAGUtCHquo3pUoGkZKngHgsUfd1ZJbpYPf13rUypW+6SRbxF0=g@mail.gmail.com>
	<53331BC0.1020205@gmail.com>
	<CAGUtCHop9RYWWg+v4W95S0wj4SB64F1sjK-BqaPGzbsv0bXNzg@mail.gmail.com>
	<533336BB.7000403@gmail.com>
	<533356AD.2090503@gmail.com>
	<5334B8E8.1080205@gmail.com>
	<CAGHyZ6K01-UCzL0bWBT2PV+VrkkSWoxny-WE25veC2shRcpX0g@mail.gmail.com>
	<CAD-fFUJzuf2j8aWhmyY1bsYOr_qdSzfemvYAo_1W9G6CFbNpag@mail.gmail.com>
	<5335A23A.6070204@gmail.com>
	<CAL5zq9YwFH8pRxhFpimqsqCFaSonO7JOJCDLrJniT4stUTSv8A@mail.gmail.com>
	<CAOyNEs7Dm=VAPx13rftNrf5KHvxRaW_pBpv7LNG4CF=0D1i4uA@mail.gmail.com>
	<CAL5zq9bpf_migPAP3nvRaN-NgkR_RRqVK7qhnTKUobo5xNCSmg@mail.gmail.com>
	<CAGHyZ6+BggBP8j-26taYsoxZ0DFxn3z0qxvNRGZUH74f6rtzRA@mail.gmail.com>
	<CAL5zq9YdBo7bWxgn9_BGF+dQD3dbmDK-1LbRVqTg5zr4XFK1aQ@mail.gmail.com>
Date: Fri, 28 Mar 2014 18:10:52 -0700
Message-ID: 
 <CAOyNEs5-YMS57RR4puZ13o4F=NM-ywbA8CkrLutfLODj=6zncw@mail.gmail.com>
Subject: Re: [DISCUSS] MiniAccumuloCluster goals and approach
From: Josh Elser <josh.elser@gmail.com>
To: dev@accumulo.apache.org
Content-Type: multipart/alternative; boundary=f46d0447a05153502c04f5b47d8a

--f46d0447a05153502c04f5b47d8a
Content-Type: text/plain; charset=UTF-8

I'll have something integrated to 1.6.0 this weekend. Something concrete
may help to firm everyone's opinion.
On Mar 28, 2014 5:54 PM, "Christopher" <ctubbsii@apache.org> wrote:

> If it's marked Experimental in the javadocs, I think it may be fine.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Fri, Mar 28, 2014 at 4:50 PM, Sean Busbey <busbey+lists@cloudera.com>
> wrote:
> > If the new interface is not in the public API, then I think adding it
> > (without deprecating MAC) is fine.
> >
> > That way it can evolve if needed and we can add it to the public API on a
> > later release.
> >
> >
> > On Fri, Mar 28, 2014 at 3:39 PM, Christopher <ctubbsii@apache.org>
> wrote:
> >
> >> But... without more time to fully develop the requirements for the
> >> interface, with a few implementations, it's probably going to change
> >> anyway. I think even adding the interface could complicate the
> >> follow-on work. But... *shrug*.... maybe you can have guarantees that
> >> the interface will stay as is (same package, same methods, same name,
> >> etc.)?
> >>
> >> --
> >> Christopher L Tubbs II
> >> http://gravatar.com/ctubbsii
> >>
> >>
> >> On Fri, Mar 28, 2014 at 3:14 PM, Josh Elser <josh.elser@gmail.com>
> wrote:
> >> > Not even the addition of a new interface, Christopher? I'd very much
> like
> >> > to have an interface that we can get in 1.6.0 at a minimum. I wouldn't
> >> even
> >> > push for any deprecation of what's currently in place.
> >> > On Mar 28, 2014 10:02 AM, "Christopher" <ctubbsii@apache.org> wrote:
> >> >
> >> >> I don't think any of this should be done for 1.6.0, but I like the
> >> >> idea of creating a separate cluster interface for testing. I think it
> >> >> should be integrated into the accumulo-maven-plugin, also. I think
> the
> >> >> idea should be hammered out, and tested as a separate thing, to
> >> >> experiment with the options, and provided as a complete feature for
> >> >> the next major release. If it would change packaging dependencies, it
> >> >> shouldn't even be done for 1.6.x bugfix releases.
> >> >>
> >> >> --
> >> >> Christopher L Tubbs II
> >> >> http://gravatar.com/ctubbsii
> >> >>
> >> >>
> >> >> On Fri, Mar 28, 2014 at 12:24 PM, Josh Elser <josh.elser@gmail.com>
> >> wrote:
> >> >> > Oh, I like that idea, Bill & Sean.
> >> >> >
> >> >> > Package: org.apache.accumulo.cluster
> >> >> > Public API: org.apache.accumulo.cluster.AccumuloCluster
> >> >> > MAC: org.apache.accumulo.cluster.mini.MiniAccumuloCluster
> (implements
> >> >> > AccumuloCluster, allows for backwards compat)
> >> >> > Yarn: org.apache.accumulo.cluster.yarn
> >> >> > Docker: ...
> >> >> > Mesos: ...
> >> >> >
> >> >> > etc etc etc.
> >> >> >
> >> >> > One question in my mind, do we keep the maven module
> >> >> 'accumulo-minicluster'?
> >> >> > I would imagine that if we struck the 'mini' portion from 1.6 that
> >> would
> >> >> > create some confusion. Would it be worth the indirection to rename
> >> >> > accumulo-minicluster to accumulo-cluster and then create a new
> >> >> > accumulo-minicluster module that depends on accumulo-minicluster
> (but
> >> >> > contains no code itself) to preserve the 1.4 and 1.5 poms to
> generally
> >> >> work
> >> >> > with a version bump? I'm not sure if Maven would be happy with that
> >> or do
> >> >> > what I think it "should".
> >> >> >
> >> >> >
> >> >> > On 3/28/14, 6:26 AM, Bill Havanki wrote:
> >> >> >>
> >> >> >> I've been watching the conversation on the side, but I wanted to
> >> mention
> >> >> >> that it seems the focus isn't so much on "mini" clusters anymore.
> >> You're
> >> >> >> thinking of programmatic cluster management, whether one node or
> >> many.
> >> >> The
> >> >> >> idea of a basic cluster management interface, with MAC as an
> >> >> >> implementation, is promising. A package name of just "cluster"
> could
> >> >> work.
> >> >> >>
> >> >> >> Carry on :)
> >> >> >>
> >> >> >> Bill H
> >> >> >>
> >> >> >>
> >> >> >> On Fri, Mar 28, 2014 at 12:39 AM, Sean Busbey
> >> >> >> <busbey+lists@cloudera.com>wrote:
> >> >> >>
> >> >> >>> If you decide to go the mapred/mapreduce way, you could go with
> the
> >> >> >>> package
> >> >> >>> name "mini".
> >> >> >>>
> >> >> >>> alternatively, we can do a multi-stage change out
> >> >> >>>
> >> >> >>> 1)  1.6.x:  introduce TestAccumuloCluster interface, @deprecate
> >> >> >>> MiniAccumuloCluster class and make it implement
> TestAccumuloCluster
> >> >> >>>
> >> >> >>> 2) 1.6 + major: change MiniAccumuloCluster to an interface that
> >> extends
> >> >> >>> TestAccumuloCluster, @deprecate TestAccumuloCluster
> >> >> >>>
> >> >> >>> 3) 1.6 + 2 major: remove TestAccumuloCluster
> >> >> >>>
> >> >> >>> Or just go with TestAccumuloCluster as the interface, have
> >> >> >>> MiniAccumuloCluster as the local pseudo distributed
> implementation,
> >> and
> >> >> >>> then call your new one something like YarnAccumuloCluster.
> >> >> >>>
> >> >> >>> In that case we could use the deprecation cycle to move the MAC
> >> class
> >> >> out
> >> >> >>> of the public api.
> >> >> >>>
> >> >> >>>
> >> >> >>> On Thu, Mar 27, 2014 at 6:48 PM, Josh Elser <
> josh.elser@gmail.com>
> >> >> wrote:
> >> >> >>>
> >> >> >>>> Thoughts on if this would be an acceptable change for 1.6.0 to
> >> >> alleviate
> >> >> >>>> future cruft?
> >> >> >>>>
> >> >> >>>> Suggestions on the new package and/or class name would be
> greatly
> >> >> >>>> appreciated over "NewMiniAccumuloC*".
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> On 3/26/14, 3:37 PM, Josh Elser wrote:
> >> >> >>>>
> >> >> >>>>> Those who are interested: check out
> >> >> >>>>> https://github.com/joshelser/accumulo/commit/
> >> >> >>>>> 9f63cf32559ab514a69ff2c6b02acef9c9cbb4e8
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> tl;dr I could create some real interfaces for the cluster and
> >> config,
> >> >> >>>>> which are "hidden" under the covers by the 1.4 and 1.5
> >> >> >>>>> MiniAccumuloCluster and MiniAccumuloConfig classes. This
> >> de-couples
> >> >> the
> >> >> >>>>> default implementation, gives us the ability to hide
> >> "implementation
> >> >> >>>>> details" if wanted, and moves us towards some factory methods
> >> instead
> >> >> >>>>> of
> >> >> >>>>> calling a class directly.
> >> >> >>>>>
> >> >> >>>>> Thoughts?
> >> >> >>>>>
> >> >> >>>>> On 3/26/14, 1:21 PM, Josh Elser wrote:
> >> >> >>>>>
> >> >> >>>>>> Yes, very much experimental at this point.
> >> >> >>>>>>
> >> >> >>>>>> What I'm most concerned about is having reasonable hooks up
> >> front,
> >> >> not
> >> >> >>>>>> trying to make an implementation for inclusion 1.6.0.
> >> >> >>>>>>
> >> >> >>>>>> Regarding additions, the implementations already contains most
> >> >> things
> >> >> >>>>>> I
> >> >> >>>>>> would want to expose. I haven't come up with anything that
> would
> >> be
> >> >> >>>>>> generally returned through the "API" rather than through this
> >> >> proposed
> >> >> >>>>>> implementation (e.g. YARN connection information)
> >> >> >>>>>>
> >> >> >>>>>> On 3/26/14, 11:57 AM, Keith Turner wrote:
> >> >> >>>>>>
> >> >> >>>>>>> What you are trying to do sounds interesting.  It also sounds
> >> >> >>>>>>> experimental
> >> >> >>>>>>> and in the early stages.   Is there anything specific you
> think
> >> >> >>>>>>> should be
> >> >> >>>>>>> done for 1.6.0 w/ regards to MAC API?
> >> >> >>>>>>>
> >> >> >>>>>>>
> >> >> >>>>>>> On Wed, Mar 26, 2014 at 2:26 PM, Josh Elser <
> >> josh.elser@gmail.com>
> >> >> >>>>>>> wrote:
> >> >> >>>>>>>
> >> >> >>>>>>>   On 3/26/14, 11:13 AM, Keith Turner wrote:
> >> >> >>>>>>>>
> >> >> >>>>>>>>
> >> >> >>>>>>>>   On Wed, Mar 26, 2014 at 2:05 PM, Josh Elser <
> >> >> josh.elser@gmail.com>
> >> >> >>>>>>>>>
> >> >> >>>>>>>>> wrote:
> >> >> >>>>>>>>>
> >> >> >>>>>>>>>    On 3/26/14, 10:57 AM, Keith Turner wrote:
> >> >> >>>>>>>>>
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>    Can you give an example of what you are thinking of? I
> >> don't
> >> >> >>>>>>>>>> understand
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>> you
> >> >> >>>>>>>>>>> viewpoint either
> >> >> >>>>>>>>>>>
> >> >> >>>>>>>>>>>
> >> >> >>>>>>>>>>>   Sure. One limitation of MAC, in general as a testing
> >> harness,
> >> >> >>>>>>>>>>> is
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> that it
> >> >> >>>>>>>>>> doesn't adequately exercise multi-node implementations.
> You
> >> can
> >> >> >>>>>>>>>> run
> >> >> >>>>>>>>>> multiple tservers, but they are all on the same host which
> >> >> limits
> >> >> >>>
> >> >> >>> the
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> validity of a "robust" test. This is my immediate goal.
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> Multi-node deployments are capable using something like
> >> Mesos or
> >> >> >>>>>>>>>> Yarn.
> >> >> >>>>>>>>>> Given that there is already functioning support to deploy
> >> >> Accumulo
> >> >> >>>
> >> >> >>> on
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> Yarn,
> >> >> >>>>>>>>>> this was my goal.
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>> My goal is to be able to have the ability to run all of
> our
> >> >> >>>>>>>>>> AbstractMacIT
> >> >> >>>>>>>>>> implementations against "real" hardware without changing a
> >> >> single
> >> >> >>>>>>>>>> line of
> >> >> >>>>>>>>>> test code (ok - maybe a line or two to do injection of the
> >> MAC
> >> >> >>>>>>>>>> implementation). The point is, I believe there could be a
> >> huge
> >> >> >>>>>>>>>> testing
> >> >> >>>>>>>>>> gain
> >> >> >>>>>>>>>> from being able to write tests which leverage yarn, have
> the
> >> >> same
> >> >> >>>>>>>>>> programmatic configuration API from MAC, and provide near
> >> "real"
> >> >> >>>>>>>>>> Accumulo
> >> >> >>>>>>>>>> semantics.
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>
> >> >> >>>>>>>>>>   Ok so you want to MAC to be an interface so that you can
> >> >> provide
> >> >> >>>>>>>>>> a
> >> >> >>>>>>>>>
> >> >> >>>>>>>>> completely different implementation?
> >> >> >>>>>>>>>
> >> >> >>>>>>>>>
> >> >> >>>>>>>>>   Correct. Some things would serve well in a common
> abstract
> >> base
> >> >> >>>
> >> >> >>> (e.g.
> >> >> >>>>>>>>
> >> >> >>>>>>>> numTservers, siteXml configuration), but all the nonsense
> about
> >> >> >>>>>>>> creating
> >> >> >>>>>>>> directory structures and managing Processes is
> implementation
> >> >> >>>
> >> >> >>> specific.
> >> >> >>>>>>>>
> >> >> >>>>>>>>
> >> >> >>>>>>>> Perhaps I could create a new interface that the current
> >> >> >>>
> >> >> >>> implementation
> >> >> >>>>>>>>
> >> >> >>>>>>>> implements which still provides the same semantics from 1.4
> and
> >> >> 1.5.
> >> >> >>>>>>>> Let me
> >> >> >>>>>>>> see if I can mock up what I'm thinking -- that will
> probably be
> >> >> >>>>>>>> easier than
> >> >> >>>>>>>> me trying to write it out.
> >> >> >>>>>>>>
> >> >> >>>>>>>>
> >> >> >>>>>>>
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >
> >> >>
> >>
>

--f46d0447a05153502c04f5b47d8a--