mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dharmesh Kakadia <dhkaka...@gmail.com>
Subject Re: Framework testing in Mesos
Date Tue, 14 Oct 2014 16:36:35 GMT
Thanks to both of you.

@David Idempotence (and functional style) will both mitigate the issue of
testing.

@Sharma #3 looks impressive and I hear the pain. Few questions:
* Since you already have the state machine modeling, can't the scheduler
actions also be modeled as a state machine transitions?
* Having a spec for (in form of state machine or otherwise) scheduler looks
important (and hard) goal. Mocking looks like a good thing. Is mocking
general enough to become a library available to all, to enable *verifiably*
correct scheduler behavior?

Again thanks for sharing your thoughts.

Thanks,
Dharmesh

On Mon, Oct 13, 2014 at 7:29 AM, David Greenberg <dsg123456789@gmail.com>
wrote:

> Specifically with regards to the state of the framework due to callback
> ordering, we ensure that our framework is written in a functional style, so
> that all callbacks atomically transform the previous state to a new state.
> By doing this, we serialize all callbacks. At this point, you can do
> generative testing to create events and run them through your system. This,
> at least, makes #3 possible.
>
> For #4, we are pretty careful to choose idempotent writes into the DB and
> a DB that supports snapshot reads. This way, you can just use at-least-once
> semantics for easy-to-implement retries. If a write fails, you just crash,
> since that means your DB's completely down. Then we test by thinking
> through and discussing whether operations have this idempotency property
> and the simple retry logic independently. This starts to get at a way to
> manage #4 to avoid learning in production.
>
> On Sun, Oct 12, 2014 at 11:44 AM, Dharmesh Kakadia <dhkakadia@gmail.com>
> wrote:
>
>> Thanks David.
>>
>> Taking state of the framework is an interesting design. I am assuming the
>> scheduler is maintaining the state and then handing tasks on slaves. If
>> that's the case, we can safely test executor (stateless - receiving event
>> and returning appropriate status to the scheduler). You construct scheduler
>> tests similarly by passing different states and event and observing the
>> next state. This way you will be sure that your callbacks works fine in
>> *isolation*. I would be concerned about the state of the framework in
>> case of callback ordering (or re-execution) in *all possible scenarios*.
>> Mocking is exactly what might uncover such bugs, but as you pointed out, I
>> also think it would not be trivial for many frameworks.
>>
>> At a high-level it would be important to know for frameworks developers
>> that,
>> 1. executors are working fine in isolation on fresh start, implementing
>> the feature.
>> 2. executors are working fine when rescheduled/restarted/in presence of
>> other executors.
>> 3. scheduler is working fine in isolation.
>> 4. scheduler is fine in the wild ( in presence of
>> others/failures/checkpointing/...).
>>
>> 1 is easy to do traditionally. 2 is possible if your executors do not
>> have side effect or using Docker etc.
>> 3 and 4 are not easy to do. I think having support/library for testing
>> scheduler is something all the framework writer would benefit from. Not
>> having to think about communication between executors and scheduler is
>> already a big plus, can we also make it easier for developers to test about
>> their scheduler behaviour?
>>
>> Thoughts?
>>
>> I would love to hear thoughts from others.
>>
>> Thanks,
>> Dharmesh
>>
>> On Sun, Oct 12, 2014 at 8:03 PM, David Greenberg <dsg123456789@gmail.com>
>> wrote:
>>
>>> For our frameworks, we don't tend to do much automated testing of the
>>> Mesos interface--instead, we construct the framework state, then "send it a
>>> message", since our callbacks take the state of the framework + the event
>>> as the argument. This way, we don't need to have mesos running, and we can
>>> trim away large amounts of code necessary to connect to mesos but
>>> unnecessary for the actual feature under test. We've also been
>>> experimenting with simulation testing by mocking out the mesos APIs. These
>>> techniques are mostly effective when you can pretend that the executors
>>> you're using don't communicate much, or when they're trivial to mock.
>>>
>>> On Sun, Oct 12, 2014 at 9:42 AM, Dharmesh Kakadia <dhkakadia@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am working on a tiny experimental framework for Mesos. I was
>>>> wondering what is the recommended way of writing testcases for framework
>>>> testing. I looked at the several existing frameworks, but its still not
>>>> clear to me. I understand that I might be able to test executor
>>>> functionality in isolation through normal test cases, but testing as a
>>>> whole framework is what I am unclear about.
>>>>
>>>> Suggestions? Is that a non-goal? How do other framework developers go
>>>> about it?
>>>>
>>>> Also, on the related note, is there a way to debug frameworks in better
>>>> way than sifting through logs?
>>>>
>>>> Thanks,
>>>> Dharmesh
>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message