accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: [DISCUSS] MiniAccumuloCluster goals and approach
Date Wed, 26 Mar 2014 22:37:33 GMT
Those who are interested: check out 
https://github.com/joshelser/accumulo/commit/9f63cf32559ab514a69ff2c6b02acef9c9cbb4e8

tl;dr I could create some real interfaces for the cluster and config, 
which are "hidden" under the covers by the 1.4 and 1.5 
MiniAccumuloCluster and MiniAccumuloConfig classes. This de-couples the 
default implementation, gives us the ability to hide "implementation 
details" if wanted, and moves us towards some factory methods instead of 
calling a class directly.

Thoughts?

On 3/26/14, 1:21 PM, Josh Elser wrote:
> Yes, very much experimental at this point.
>
> What I'm most concerned about is having reasonable hooks up front, not
> trying to make an implementation for inclusion 1.6.0.
>
> Regarding additions, the implementations already contains most things I
> would want to expose. I haven't come up with anything that would be
> generally returned through the "API" rather than through this proposed
> implementation (e.g. YARN connection information)
>
> On 3/26/14, 11:57 AM, Keith Turner wrote:
>> What you are trying to do sounds interesting.  It also sounds
>> experimental
>> and in the early stages.   Is there anything specific you think should be
>> done for 1.6.0 w/ regards to MAC API?
>>
>>
>> On Wed, Mar 26, 2014 at 2:26 PM, Josh Elser <josh.elser@gmail.com> wrote:
>>
>>> On 3/26/14, 11:13 AM, Keith Turner wrote:
>>>
>>>> On Wed, Mar 26, 2014 at 2:05 PM, Josh Elser <josh.elser@gmail.com>
>>>> wrote:
>>>>
>>>>   On 3/26/14, 10:57 AM, Keith Turner wrote:
>>>>>
>>>>>   Can you give an example of what you are thinking of? I don't
>>>>> understand
>>>>>> you
>>>>>> viewpoint either
>>>>>>
>>>>>>
>>>>> Sure. One limitation of MAC, in general as a testing harness, is
>>>>> that it
>>>>> doesn't adequately exercise multi-node implementations. You can run
>>>>> multiple tservers, but they are all on the same host which limits the
>>>>> validity of a "robust" test. This is my immediate goal.
>>>>>
>>>>> Multi-node deployments are capable using something like Mesos or Yarn.
>>>>> Given that there is already functioning support to deploy Accumulo on
>>>>> Yarn,
>>>>> this was my goal.
>>>>>
>>>>> My goal is to be able to have the ability to run all of our
>>>>> AbstractMacIT
>>>>> implementations against "real" hardware without changing a single
>>>>> line of
>>>>> test code (ok - maybe a line or two to do injection of the MAC
>>>>> implementation). The point is, I believe there could be a huge testing
>>>>> gain
>>>>> from being able to write tests which leverage yarn, have the same
>>>>> programmatic configuration API from MAC, and provide near "real"
>>>>> Accumulo
>>>>> semantics.
>>>>>
>>>>>
>>>> Ok so you want to MAC to be an interface so that you can provide a
>>>> completely different implementation?
>>>>
>>>>
>>> Correct. Some things would serve well in a common abstract base (e.g.
>>> numTservers, siteXml configuration), but all the nonsense about creating
>>> directory structures and managing Processes is implementation specific.
>>>
>>> Perhaps I could create a new interface that the current implementation
>>> implements which still provides the same semantics from 1.4 and 1.5.
>>> Let me
>>> see if I can mock up what I'm thinking -- that will probably be
>>> easier than
>>> me trying to write it out.
>>>
>>

Mime
View raw message