Mailing-List: contact dev-help@cordova.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cordova.apache.org
Received-SPF: pass (athena.apache.org: domain of anis.kadri@gmail.com
 designates 209.85.192.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAMk5LLk8Fwx+Efcn2zPLnjj2sb9wfDBLYaFrS9w7=7822a9n-w@mail.gmail.com>
References: 
 <CAMk5LLk6j1rqSHoTSqjyPYC=ePrHEycV2R-UuzhwCw8fyHKKnQ@mail.gmail.com>
	<CAEeF2Td4OYyEYukBYAP5fxDOm43HV3Eq_kE5jfFygQHsKr6JBw@mail.gmail.com>
	<CAEeF2TemAVgT-=+bE5W3Lti10EJmGHqvBnjXZ7tUFjGPR44zdg@mail.gmail.com>
	<299C48FD-159B-4A4F-9877-F2AF6105714C@gmail.com>
	<CAMk5LLk8Fwx+Efcn2zPLnjj2sb9wfDBLYaFrS9w7=7822a9n-w@mail.gmail.com>
Date: Thu, 7 Nov 2013 11:57:49 -0800
Message-ID: 
 <CANpVHHxkvK4A0xsSMpajJjcJtzw5oXy59JOjmMLbD4AVURREUA@mail.gmail.com>
Subject: Re: Refactoring the CLI tests
From: Anis KADRI <anis.kadri@gmail.com>
To: "dev@cordova.apache.org" <dev@cordova.apache.org>
Cc: Michal Mocny <mmocny@chromium.org>
Content-Type: text/plain; charset=ISO-8859-1

The reason why every FS call is mocked is speed but speed is
subjective in my opinion. Given the features of CLI, my opinion is
that anything < 1 minute is acceptable. When I run tests, I am not
actively watching them execute.
I think the only calls that should be mocked are network calls because
you should be able to run tests offline.
I think we should convert (trash?) existing tests back to what they
were instead of adding more end-to-end tests. We should also pay more
attention de Windows.

On Thu, Nov 7, 2013 at 10:31 AM, Braden Shepherdson <braden@chromium.org> wrote:
> Alright, Mark and I have discussed this further, and we will be beginning
> the effort with some end-to-end tests that will supplement the existing
> tests.
>
> To some extent this is duplicating things that go on in the CI, since it
> checks out various plugins using the tools. But we think it's still a
> worthwhile effort since it will make it much easier to catch any problems
> at commit time.
>
> The main pain point in the existing tests that we'll want to fix is their
> fragility. The worst offenders here are the platform parsers, especially
> wp7+8. So Steve, if you're looking for ways to help, I'd suggest looking at
> those (wp in particular, but all of them). These are some of the worst,
> most vacuous tests we have. They're not providing sample inputs and
> checking the outputs, they are checking that the right functions get
> called. I propose that they should be operating on a real, checked-in,
> example project, in the spec/fixtures directory. The tests should run the
> parsers, and make sure all the files are in the right places and have the
> right contents.
>
> Braden
>
>
> On Thu, Nov 7, 2013 at 1:02 PM, Steve Gill <stevengill97@gmail.com> wrote:
>
>> I don't think we should scrap the current tests. I am totally in favor of
>> having new end to end tests. We need to catch regressions better.
>>
>> Braden, let me know how I can help.
>>
>>
>>
>> > On Nov 7, 2013, at 9:02 AM, Michal Mocny <mmocny@chromium.org> wrote:
>> >
>> > Discussing locally with Braden.. these tests seem to be testing internal
>> > details instead of expected functionality.  Its quite common for valid
>> > patches to break the tests and invalid patches to leave the tests
>> passing.
>> > There have been several occurrences recently where things landed even
>> > though "cordova create" or "plugin add" were hosed, which seems fairly
>> > fundamental to keep working ;)
>> >
>> >
>> >> On Thu, Nov 7, 2013 at 11:57 AM, Michal Mocny <mmocny@chromium.org>
>> wrote:
>> >>
>> >> +1 to testing end-to-end, but I thought thats what BuildBot/Medic does?
>> >> Likely we want to ship those end-to-end test scripts so users can test
>> >> changes locally, but I think the tests are are inside CLI now are meant
>> to
>> >> be unit tests, which yes, have very limited usefulness in isolation, but
>> >> perhaps shouldn't be entirely replaced?  If they really are useless (not
>> >> sure) then they should be fixed/removed.
>> >>
>> >>
>> >> On Thu, Nov 7, 2013 at 11:40 AM, Braden Shepherdson <
>> braden@chromium.org>wrote:
>> >>
>> >>> The CLI tests are bad. I propose making them better.
>> >>>
>> >>> The tests are bad for two reasons:
>> >>> 1. They're fragile because the tests depend on exactly the right
>> functions
>> >>> being called, sometimes in the right order.
>> >>> 2. They don't test what we really want, which is that projects get
>> created
>> >>> and all the files are in the right places.
>> >>>
>> >>> I propose letting the tests actually run filesystem and related calls,
>> >>> instead of always mocking them out. In the simplest form, that means
>> >>> running them on the real filesystem. If that's too slow, we can
>> >>> investigate
>> >>> other alternatives, like using a ramdisk, or using that emulated fs
>> that
>> >>> runs everything in RAM inside Node.
>> >>>
>> >>> Mark and I are planning to work on this, starting with the CLI tests.
>> The
>> >>> Plugman tests are better in this regard, but could probably stand to be
>> >>> really called as well.
>> >>>
>> >>> Any thoughts or objections?
>> >>>
>> >>> Braden
>> >>
>> >>
>>