accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher <ctubb...@apache.org>
Subject Re: [DISCUSS] MiniAccumuloCluster goals and approach
Date Fri, 28 Mar 2014 21:54:02 GMT
If it's marked Experimental in the javadocs, I think it may be fine.

--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Fri, Mar 28, 2014 at 4:50 PM, Sean Busbey <busbey+lists@cloudera.com> wrote:
> If the new interface is not in the public API, then I think adding it
> (without deprecating MAC) is fine.
>
> That way it can evolve if needed and we can add it to the public API on a
> later release.
>
>
> On Fri, Mar 28, 2014 at 3:39 PM, Christopher <ctubbsii@apache.org> wrote:
>
>> But... without more time to fully develop the requirements for the
>> interface, with a few implementations, it's probably going to change
>> anyway. I think even adding the interface could complicate the
>> follow-on work. But... *shrug*.... maybe you can have guarantees that
>> the interface will stay as is (same package, same methods, same name,
>> etc.)?
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Fri, Mar 28, 2014 at 3:14 PM, Josh Elser <josh.elser@gmail.com> wrote:
>> > Not even the addition of a new interface, Christopher? I'd very much like
>> > to have an interface that we can get in 1.6.0 at a minimum. I wouldn't
>> even
>> > push for any deprecation of what's currently in place.
>> > On Mar 28, 2014 10:02 AM, "Christopher" <ctubbsii@apache.org> wrote:
>> >
>> >> I don't think any of this should be done for 1.6.0, but I like the
>> >> idea of creating a separate cluster interface for testing. I think it
>> >> should be integrated into the accumulo-maven-plugin, also. I think the
>> >> idea should be hammered out, and tested as a separate thing, to
>> >> experiment with the options, and provided as a complete feature for
>> >> the next major release. If it would change packaging dependencies, it
>> >> shouldn't even be done for 1.6.x bugfix releases.
>> >>
>> >> --
>> >> Christopher L Tubbs II
>> >> http://gravatar.com/ctubbsii
>> >>
>> >>
>> >> On Fri, Mar 28, 2014 at 12:24 PM, Josh Elser <josh.elser@gmail.com>
>> wrote:
>> >> > Oh, I like that idea, Bill & Sean.
>> >> >
>> >> > Package: org.apache.accumulo.cluster
>> >> > Public API: org.apache.accumulo.cluster.AccumuloCluster
>> >> > MAC: org.apache.accumulo.cluster.mini.MiniAccumuloCluster (implements
>> >> > AccumuloCluster, allows for backwards compat)
>> >> > Yarn: org.apache.accumulo.cluster.yarn
>> >> > Docker: ...
>> >> > Mesos: ...
>> >> >
>> >> > etc etc etc.
>> >> >
>> >> > One question in my mind, do we keep the maven module
>> >> 'accumulo-minicluster'?
>> >> > I would imagine that if we struck the 'mini' portion from 1.6 that
>> would
>> >> > create some confusion. Would it be worth the indirection to rename
>> >> > accumulo-minicluster to accumulo-cluster and then create a new
>> >> > accumulo-minicluster module that depends on accumulo-minicluster (but
>> >> > contains no code itself) to preserve the 1.4 and 1.5 poms to generally
>> >> work
>> >> > with a version bump? I'm not sure if Maven would be happy with that
>> or do
>> >> > what I think it "should".
>> >> >
>> >> >
>> >> > On 3/28/14, 6:26 AM, Bill Havanki wrote:
>> >> >>
>> >> >> I've been watching the conversation on the side, but I wanted to
>> mention
>> >> >> that it seems the focus isn't so much on "mini" clusters anymore.
>> You're
>> >> >> thinking of programmatic cluster management, whether one node or
>> many.
>> >> The
>> >> >> idea of a basic cluster management interface, with MAC as an
>> >> >> implementation, is promising. A package name of just "cluster"
could
>> >> work.
>> >> >>
>> >> >> Carry on :)
>> >> >>
>> >> >> Bill H
>> >> >>
>> >> >>
>> >> >> On Fri, Mar 28, 2014 at 12:39 AM, Sean Busbey
>> >> >> <busbey+lists@cloudera.com>wrote:
>> >> >>
>> >> >>> If you decide to go the mapred/mapreduce way, you could go
with the
>> >> >>> package
>> >> >>> name "mini".
>> >> >>>
>> >> >>> alternatively, we can do a multi-stage change out
>> >> >>>
>> >> >>> 1)  1.6.x:  introduce TestAccumuloCluster interface, @deprecate
>> >> >>> MiniAccumuloCluster class and make it implement TestAccumuloCluster
>> >> >>>
>> >> >>> 2) 1.6 + major: change MiniAccumuloCluster to an interface
that
>> extends
>> >> >>> TestAccumuloCluster, @deprecate TestAccumuloCluster
>> >> >>>
>> >> >>> 3) 1.6 + 2 major: remove TestAccumuloCluster
>> >> >>>
>> >> >>> Or just go with TestAccumuloCluster as the interface, have
>> >> >>> MiniAccumuloCluster as the local pseudo distributed implementation,
>> and
>> >> >>> then call your new one something like YarnAccumuloCluster.
>> >> >>>
>> >> >>> In that case we could use the deprecation cycle to move the
MAC
>> class
>> >> out
>> >> >>> of the public api.
>> >> >>>
>> >> >>>
>> >> >>> On Thu, Mar 27, 2014 at 6:48 PM, Josh Elser <josh.elser@gmail.com>
>> >> wrote:
>> >> >>>
>> >> >>>> Thoughts on if this would be an acceptable change for 1.6.0
to
>> >> alleviate
>> >> >>>> future cruft?
>> >> >>>>
>> >> >>>> Suggestions on the new package and/or class name would
be greatly
>> >> >>>> appreciated over "NewMiniAccumuloC*".
>> >> >>>>
>> >> >>>>
>> >> >>>> On 3/26/14, 3:37 PM, Josh Elser wrote:
>> >> >>>>
>> >> >>>>> Those who are interested: check out
>> >> >>>>> https://github.com/joshelser/accumulo/commit/
>> >> >>>>> 9f63cf32559ab514a69ff2c6b02acef9c9cbb4e8
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> tl;dr I could create some real interfaces for the cluster
and
>> config,
>> >> >>>>> which are "hidden" under the covers by the 1.4 and
1.5
>> >> >>>>> MiniAccumuloCluster and MiniAccumuloConfig classes.
This
>> de-couples
>> >> the
>> >> >>>>> default implementation, gives us the ability to hide
>> "implementation
>> >> >>>>> details" if wanted, and moves us towards some factory
methods
>> instead
>> >> >>>>> of
>> >> >>>>> calling a class directly.
>> >> >>>>>
>> >> >>>>> Thoughts?
>> >> >>>>>
>> >> >>>>> On 3/26/14, 1:21 PM, Josh Elser wrote:
>> >> >>>>>
>> >> >>>>>> Yes, very much experimental at this point.
>> >> >>>>>>
>> >> >>>>>> What I'm most concerned about is having reasonable
hooks up
>> front,
>> >> not
>> >> >>>>>> trying to make an implementation for inclusion
1.6.0.
>> >> >>>>>>
>> >> >>>>>> Regarding additions, the implementations already
contains most
>> >> things
>> >> >>>>>> I
>> >> >>>>>> would want to expose. I haven't come up with anything
that would
>> be
>> >> >>>>>> generally returned through the "API" rather than
through this
>> >> proposed
>> >> >>>>>> implementation (e.g. YARN connection information)
>> >> >>>>>>
>> >> >>>>>> On 3/26/14, 11:57 AM, Keith Turner wrote:
>> >> >>>>>>
>> >> >>>>>>> What you are trying to do sounds interesting.
 It also sounds
>> >> >>>>>>> experimental
>> >> >>>>>>> and in the early stages.   Is there anything
specific you think
>> >> >>>>>>> should be
>> >> >>>>>>> done for 1.6.0 w/ regards to MAC API?
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> On Wed, Mar 26, 2014 at 2:26 PM, Josh Elser
<
>> josh.elser@gmail.com>
>> >> >>>>>>> wrote:
>> >> >>>>>>>
>> >> >>>>>>>   On 3/26/14, 11:13 AM, Keith Turner wrote:
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>>   On Wed, Mar 26, 2014 at 2:05 PM, Josh
Elser <
>> >> josh.elser@gmail.com>
>> >> >>>>>>>>>
>> >> >>>>>>>>> wrote:
>> >> >>>>>>>>>
>> >> >>>>>>>>>    On 3/26/14, 10:57 AM, Keith Turner
wrote:
>> >> >>>>>>>>>
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>    Can you give an example of what
you are thinking of? I
>> don't
>> >> >>>>>>>>>> understand
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>> you
>> >> >>>>>>>>>>> viewpoint either
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>>   Sure. One limitation of MAC,
in general as a testing
>> harness,
>> >> >>>>>>>>>>> is
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> that it
>> >> >>>>>>>>>> doesn't adequately exercise multi-node
implementations. You
>> can
>> >> >>>>>>>>>> run
>> >> >>>>>>>>>> multiple tservers, but they are
all on the same host which
>> >> limits
>> >> >>>
>> >> >>> the
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> validity of a "robust" test. This
is my immediate goal.
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Multi-node deployments are capable
using something like
>> Mesos or
>> >> >>>>>>>>>> Yarn.
>> >> >>>>>>>>>> Given that there is already functioning
support to deploy
>> >> Accumulo
>> >> >>>
>> >> >>> on
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Yarn,
>> >> >>>>>>>>>> this was my goal.
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> My goal is to be able to have the
ability to run all of our
>> >> >>>>>>>>>> AbstractMacIT
>> >> >>>>>>>>>> implementations against "real"
hardware without changing a
>> >> single
>> >> >>>>>>>>>> line of
>> >> >>>>>>>>>> test code (ok - maybe a line or
two to do injection of the
>> MAC
>> >> >>>>>>>>>> implementation). The point is,
I believe there could be a
>> huge
>> >> >>>>>>>>>> testing
>> >> >>>>>>>>>> gain
>> >> >>>>>>>>>> from being able to write tests
which leverage yarn, have the
>> >> same
>> >> >>>>>>>>>> programmatic configuration API
from MAC, and provide near
>> "real"
>> >> >>>>>>>>>> Accumulo
>> >> >>>>>>>>>> semantics.
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>   Ok so you want to MAC to be an
interface so that you can
>> >> provide
>> >> >>>>>>>>>> a
>> >> >>>>>>>>>
>> >> >>>>>>>>> completely different implementation?
>> >> >>>>>>>>>
>> >> >>>>>>>>>
>> >> >>>>>>>>>   Correct. Some things would serve
well in a common abstract
>> base
>> >> >>>
>> >> >>> (e.g.
>> >> >>>>>>>>
>> >> >>>>>>>> numTservers, siteXml configuration), but
all the nonsense about
>> >> >>>>>>>> creating
>> >> >>>>>>>> directory structures and managing Processes
is implementation
>> >> >>>
>> >> >>> specific.
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>> Perhaps I could create a new interface
that the current
>> >> >>>
>> >> >>> implementation
>> >> >>>>>>>>
>> >> >>>>>>>> implements which still provides the same
semantics from 1.4 and
>> >> 1.5.
>> >> >>>>>>>> Let me
>> >> >>>>>>>> see if I can mock up what I'm thinking
-- that will probably be
>> >> >>>>>>>> easier than
>> >> >>>>>>>> me trying to write it out.
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >>
>>

Mime
View raw message