cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Braden Shepherdson <bra...@chromium.org>
Subject Re: Refactoring the CLI tests
Date Thu, 07 Nov 2013 18:31:19 GMT
Alright, Mark and I have discussed this further, and we will be beginning
the effort with some end-to-end tests that will supplement the existing
tests.

To some extent this is duplicating things that go on in the CI, since it
checks out various plugins using the tools. But we think it's still a
worthwhile effort since it will make it much easier to catch any problems
at commit time.

The main pain point in the existing tests that we'll want to fix is their
fragility. The worst offenders here are the platform parsers, especially
wp7+8. So Steve, if you're looking for ways to help, I'd suggest looking at
those (wp in particular, but all of them). These are some of the worst,
most vacuous tests we have. They're not providing sample inputs and
checking the outputs, they are checking that the right functions get
called. I propose that they should be operating on a real, checked-in,
example project, in the spec/fixtures directory. The tests should run the
parsers, and make sure all the files are in the right places and have the
right contents.

Braden


On Thu, Nov 7, 2013 at 1:02 PM, Steve Gill <stevengill97@gmail.com> wrote:

> I don't think we should scrap the current tests. I am totally in favor of
> having new end to end tests. We need to catch regressions better.
>
> Braden, let me know how I can help.
>
>
>
> > On Nov 7, 2013, at 9:02 AM, Michal Mocny <mmocny@chromium.org> wrote:
> >
> > Discussing locally with Braden.. these tests seem to be testing internal
> > details instead of expected functionality.  Its quite common for valid
> > patches to break the tests and invalid patches to leave the tests
> passing.
> > There have been several occurrences recently where things landed even
> > though "cordova create" or "plugin add" were hosed, which seems fairly
> > fundamental to keep working ;)
> >
> >
> >> On Thu, Nov 7, 2013 at 11:57 AM, Michal Mocny <mmocny@chromium.org>
> wrote:
> >>
> >> +1 to testing end-to-end, but I thought thats what BuildBot/Medic does?
> >> Likely we want to ship those end-to-end test scripts so users can test
> >> changes locally, but I think the tests are are inside CLI now are meant
> to
> >> be unit tests, which yes, have very limited usefulness in isolation, but
> >> perhaps shouldn't be entirely replaced?  If they really are useless (not
> >> sure) then they should be fixed/removed.
> >>
> >>
> >> On Thu, Nov 7, 2013 at 11:40 AM, Braden Shepherdson <
> braden@chromium.org>wrote:
> >>
> >>> The CLI tests are bad. I propose making them better.
> >>>
> >>> The tests are bad for two reasons:
> >>> 1. They're fragile because the tests depend on exactly the right
> functions
> >>> being called, sometimes in the right order.
> >>> 2. They don't test what we really want, which is that projects get
> created
> >>> and all the files are in the right places.
> >>>
> >>> I propose letting the tests actually run filesystem and related calls,
> >>> instead of always mocking them out. In the simplest form, that means
> >>> running them on the real filesystem. If that's too slow, we can
> >>> investigate
> >>> other alternatives, like using a ramdisk, or using that emulated fs
> that
> >>> runs everything in RAM inside Node.
> >>>
> >>> Mark and I are planning to work on this, starting with the CLI tests.
> The
> >>> Plugman tests are better in this regard, but could probably stand to be
> >>> really called as well.
> >>>
> >>> Any thoughts or objections?
> >>>
> >>> Braden
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message