accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <busbey+li...@cloudera.com>
Subject Re: [DISCUSS] MiniAccumuloCluster goals and approach
Date Fri, 28 Mar 2014 20:50:56 GMT
If the new interface is not in the public API, then I think adding it
(without deprecating MAC) is fine.

That way it can evolve if needed and we can add it to the public API on a
later release.


On Fri, Mar 28, 2014 at 3:39 PM, Christopher <ctubbsii@apache.org> wrote:

> But... without more time to fully develop the requirements for the
> interface, with a few implementations, it's probably going to change
> anyway. I think even adding the interface could complicate the
> follow-on work. But... *shrug*.... maybe you can have guarantees that
> the interface will stay as is (same package, same methods, same name,
> etc.)?
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Fri, Mar 28, 2014 at 3:14 PM, Josh Elser <josh.elser@gmail.com> wrote:
> > Not even the addition of a new interface, Christopher? I'd very much like
> > to have an interface that we can get in 1.6.0 at a minimum. I wouldn't
> even
> > push for any deprecation of what's currently in place.
> > On Mar 28, 2014 10:02 AM, "Christopher" <ctubbsii@apache.org> wrote:
> >
> >> I don't think any of this should be done for 1.6.0, but I like the
> >> idea of creating a separate cluster interface for testing. I think it
> >> should be integrated into the accumulo-maven-plugin, also. I think the
> >> idea should be hammered out, and tested as a separate thing, to
> >> experiment with the options, and provided as a complete feature for
> >> the next major release. If it would change packaging dependencies, it
> >> shouldn't even be done for 1.6.x bugfix releases.
> >>
> >> --
> >> Christopher L Tubbs II
> >> http://gravatar.com/ctubbsii
> >>
> >>
> >> On Fri, Mar 28, 2014 at 12:24 PM, Josh Elser <josh.elser@gmail.com>
> wrote:
> >> > Oh, I like that idea, Bill & Sean.
> >> >
> >> > Package: org.apache.accumulo.cluster
> >> > Public API: org.apache.accumulo.cluster.AccumuloCluster
> >> > MAC: org.apache.accumulo.cluster.mini.MiniAccumuloCluster (implements
> >> > AccumuloCluster, allows for backwards compat)
> >> > Yarn: org.apache.accumulo.cluster.yarn
> >> > Docker: ...
> >> > Mesos: ...
> >> >
> >> > etc etc etc.
> >> >
> >> > One question in my mind, do we keep the maven module
> >> 'accumulo-minicluster'?
> >> > I would imagine that if we struck the 'mini' portion from 1.6 that
> would
> >> > create some confusion. Would it be worth the indirection to rename
> >> > accumulo-minicluster to accumulo-cluster and then create a new
> >> > accumulo-minicluster module that depends on accumulo-minicluster (but
> >> > contains no code itself) to preserve the 1.4 and 1.5 poms to generally
> >> work
> >> > with a version bump? I'm not sure if Maven would be happy with that
> or do
> >> > what I think it "should".
> >> >
> >> >
> >> > On 3/28/14, 6:26 AM, Bill Havanki wrote:
> >> >>
> >> >> I've been watching the conversation on the side, but I wanted to
> mention
> >> >> that it seems the focus isn't so much on "mini" clusters anymore.
> You're
> >> >> thinking of programmatic cluster management, whether one node or
> many.
> >> The
> >> >> idea of a basic cluster management interface, with MAC as an
> >> >> implementation, is promising. A package name of just "cluster" could
> >> work.
> >> >>
> >> >> Carry on :)
> >> >>
> >> >> Bill H
> >> >>
> >> >>
> >> >> On Fri, Mar 28, 2014 at 12:39 AM, Sean Busbey
> >> >> <busbey+lists@cloudera.com>wrote:
> >> >>
> >> >>> If you decide to go the mapred/mapreduce way, you could go with
the
> >> >>> package
> >> >>> name "mini".
> >> >>>
> >> >>> alternatively, we can do a multi-stage change out
> >> >>>
> >> >>> 1)  1.6.x:  introduce TestAccumuloCluster interface, @deprecate
> >> >>> MiniAccumuloCluster class and make it implement TestAccumuloCluster
> >> >>>
> >> >>> 2) 1.6 + major: change MiniAccumuloCluster to an interface that
> extends
> >> >>> TestAccumuloCluster, @deprecate TestAccumuloCluster
> >> >>>
> >> >>> 3) 1.6 + 2 major: remove TestAccumuloCluster
> >> >>>
> >> >>> Or just go with TestAccumuloCluster as the interface, have
> >> >>> MiniAccumuloCluster as the local pseudo distributed implementation,
> and
> >> >>> then call your new one something like YarnAccumuloCluster.
> >> >>>
> >> >>> In that case we could use the deprecation cycle to move the MAC
> class
> >> out
> >> >>> of the public api.
> >> >>>
> >> >>>
> >> >>> On Thu, Mar 27, 2014 at 6:48 PM, Josh Elser <josh.elser@gmail.com>
> >> wrote:
> >> >>>
> >> >>>> Thoughts on if this would be an acceptable change for 1.6.0
to
> >> alleviate
> >> >>>> future cruft?
> >> >>>>
> >> >>>> Suggestions on the new package and/or class name would be greatly
> >> >>>> appreciated over "NewMiniAccumuloC*".
> >> >>>>
> >> >>>>
> >> >>>> On 3/26/14, 3:37 PM, Josh Elser wrote:
> >> >>>>
> >> >>>>> Those who are interested: check out
> >> >>>>> https://github.com/joshelser/accumulo/commit/
> >> >>>>> 9f63cf32559ab514a69ff2c6b02acef9c9cbb4e8
> >> >>>>>
> >> >>>>>
> >> >>>>> tl;dr I could create some real interfaces for the cluster
and
> config,
> >> >>>>> which are "hidden" under the covers by the 1.4 and 1.5
> >> >>>>> MiniAccumuloCluster and MiniAccumuloConfig classes. This
> de-couples
> >> the
> >> >>>>> default implementation, gives us the ability to hide
> "implementation
> >> >>>>> details" if wanted, and moves us towards some factory methods
> instead
> >> >>>>> of
> >> >>>>> calling a class directly.
> >> >>>>>
> >> >>>>> Thoughts?
> >> >>>>>
> >> >>>>> On 3/26/14, 1:21 PM, Josh Elser wrote:
> >> >>>>>
> >> >>>>>> Yes, very much experimental at this point.
> >> >>>>>>
> >> >>>>>> What I'm most concerned about is having reasonable
hooks up
> front,
> >> not
> >> >>>>>> trying to make an implementation for inclusion 1.6.0.
> >> >>>>>>
> >> >>>>>> Regarding additions, the implementations already contains
most
> >> things
> >> >>>>>> I
> >> >>>>>> would want to expose. I haven't come up with anything
that would
> be
> >> >>>>>> generally returned through the "API" rather than through
this
> >> proposed
> >> >>>>>> implementation (e.g. YARN connection information)
> >> >>>>>>
> >> >>>>>> On 3/26/14, 11:57 AM, Keith Turner wrote:
> >> >>>>>>
> >> >>>>>>> What you are trying to do sounds interesting. 
It also sounds
> >> >>>>>>> experimental
> >> >>>>>>> and in the early stages.   Is there anything specific
you think
> >> >>>>>>> should be
> >> >>>>>>> done for 1.6.0 w/ regards to MAC API?
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> On Wed, Mar 26, 2014 at 2:26 PM, Josh Elser <
> josh.elser@gmail.com>
> >> >>>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>>   On 3/26/14, 11:13 AM, Keith Turner wrote:
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>   On Wed, Mar 26, 2014 at 2:05 PM, Josh Elser
<
> >> josh.elser@gmail.com>
> >> >>>>>>>>>
> >> >>>>>>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>    On 3/26/14, 10:57 AM, Keith Turner wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>    Can you give an example of what
you are thinking of? I
> don't
> >> >>>>>>>>>> understand
> >> >>>>>>>>>>
> >> >>>>>>>>>>> you
> >> >>>>>>>>>>> viewpoint either
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>   Sure. One limitation of MAC,
in general as a testing
> harness,
> >> >>>>>>>>>>> is
> >> >>>>>>>>>>
> >> >>>>>>>>>> that it
> >> >>>>>>>>>> doesn't adequately exercise multi-node
implementations. You
> can
> >> >>>>>>>>>> run
> >> >>>>>>>>>> multiple tservers, but they are all
on the same host which
> >> limits
> >> >>>
> >> >>> the
> >> >>>>>>>>>>
> >> >>>>>>>>>> validity of a "robust" test. This is
my immediate goal.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Multi-node deployments are capable
using something like
> Mesos or
> >> >>>>>>>>>> Yarn.
> >> >>>>>>>>>> Given that there is already functioning
support to deploy
> >> Accumulo
> >> >>>
> >> >>> on
> >> >>>>>>>>>>
> >> >>>>>>>>>> Yarn,
> >> >>>>>>>>>> this was my goal.
> >> >>>>>>>>>>
> >> >>>>>>>>>> My goal is to be able to have the ability
to run all of our
> >> >>>>>>>>>> AbstractMacIT
> >> >>>>>>>>>> implementations against "real" hardware
without changing a
> >> single
> >> >>>>>>>>>> line of
> >> >>>>>>>>>> test code (ok - maybe a line or two
to do injection of the
> MAC
> >> >>>>>>>>>> implementation). The point is, I believe
there could be a
> huge
> >> >>>>>>>>>> testing
> >> >>>>>>>>>> gain
> >> >>>>>>>>>> from being able to write tests which
leverage yarn, have the
> >> same
> >> >>>>>>>>>> programmatic configuration API from
MAC, and provide near
> "real"
> >> >>>>>>>>>> Accumulo
> >> >>>>>>>>>> semantics.
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>   Ok so you want to MAC to be an interface
so that you can
> >> provide
> >> >>>>>>>>>> a
> >> >>>>>>>>>
> >> >>>>>>>>> completely different implementation?
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>   Correct. Some things would serve well
in a common abstract
> base
> >> >>>
> >> >>> (e.g.
> >> >>>>>>>>
> >> >>>>>>>> numTservers, siteXml configuration), but all
the nonsense about
> >> >>>>>>>> creating
> >> >>>>>>>> directory structures and managing Processes
is implementation
> >> >>>
> >> >>> specific.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> Perhaps I could create a new interface that
the current
> >> >>>
> >> >>> implementation
> >> >>>>>>>>
> >> >>>>>>>> implements which still provides the same semantics
from 1.4 and
> >> 1.5.
> >> >>>>>>>> Let me
> >> >>>>>>>> see if I can mock up what I'm thinking -- that
will probably be
> >> >>>>>>>> easier than
> >> >>>>>>>> me trying to write it out.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message