cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Mocny <mmo...@chromium.org>
Subject Re: mobile-spec and releases: How do we test?
Date Fri, 11 Oct 2013 21:17:51 GMT
TLDR; I've implemented a plugin testing strategy that requires 0 new
tooling features, can support non-core plugins, and hopefully even support
a variety of methods for running/reporting the test results.  Super alpha
preview, but take a look if you like the direction!


NEW: CDVTest Plugin: https://github.com/mmocny/CDVTest

NEW: CordovaTest App: https://github.com/mmocny/CordovaTests

UPDATED: Converted three existing plugins to this "new style".  Code is on
feature branches of respective repos (cdvtest branch):
org.apache.cordova.device
org.apache.cordova.device-motion
org.chromium.storage
(must clone locally, switch to branch, and plugin add from local path)


Now, any plugin that wants to join in on the fun needs to provide a <js-module
src="..." name="test" /> and use this template:

exports.init = function() {
  eval(require('org.apache.cordova.test.test').injectJasmineInterface(this,
'this'));

  // TESTS GO HERE
  describe(..., function() {
    it(...);
  });
};
(The eval is optional but super useful; it will inject the jasmine
interface into your scope, so you don't have to type jasmine.describe,
jasmine.it, etc.  Not sure of any cleaner way to do this.)


STEPS:
1. create a new cordova project
2. import the CordovaTest app into your www
3. add any or all of the above plugins
4. give it a run, and try out the auto tests (all pass on ipad, some still
fail on android)

Lots of work left to do, but hopefully good enough to whet your appetite.

One thing to note, I've attempted to make CDVTest and plugin tests unaware
of the specific jasmine version the app is hosting (so that it can be
swapped without changing all plugins), but it must support the new style
interface for async tests (ie, accept a done callback).  This is the style
that node-jasmine uses, mocha uses, and jasmine-2.0 is going to use (I'm
using jasmine 2.0 rc3).  This means that core plugin tests require some
code transformations, but the net effect is cleaner tests and more common
style with our node tools' tests.

Manual tests are not implemented yet.

-Michal

On Fri, Oct 11, 2013 at 12:54 PM, Michal Mocny <mmocny@chromium.org> wrote:

> I'm throwing something together right now, actually.  I'll post my current
> progress today so you can take a look.
>
>
> On Fri, Oct 11, 2013 at 12:41 PM, Brian LeRoux <b@brian.io> wrote:
>
>> Sorry keep meaning to respond. I like Michal's first step but growing to a
>> full suite of tools. Are you currently tackling this Braden? I feel like
>> it
>> is related to the Medic stuff and maybe we should throw one of our guys on
>> the problem fully.
>>
>>
>> On Sep 27, 2013 5:10 PM, "Braden Shepherdson" <braden@chromium.org>
>> wrote:
>>
>> > Which one?
>> >
>> >
>> > On Fri, Sep 27, 2013 at 10:09 AM, Brian LeRoux <b@brian.io> wrote:
>> >
>> > > I really like your proposal as a starting point. Very simple but would
>> > > allow for in-app testing as well as on the cmd line if we so wish.
>> > >
>> > >
>> > > On Fri, Sep 27, 2013 at 3:28 PM, Michal Mocny <mmocny@chromium.org>
>> > wrote:
>> > >
>> > > > I was looking over some old emails from this list on plugin testing,
>> > and
>> > > an
>> > > > idea that was proposed way back was to ship plugin tests as a second
>> > > > plugin.  That way, you can chose to install tests, or not, and know
>> > > > explicitly if they are being copied into your final project.
>> > > >
>> > > > An alternative would be to support build targets a la
>> "release/debug"
>> > and
>> > > > have target-specific plugin.xml tags (assets, js-modules,
>> > source-file..).
>> > > >
>> > > > -Michal
>> > > >
>> > > >
>> > > > On Fri, Sep 27, 2013 at 4:52 AM, Brian LeRoux <b@brian.io> wrote:
>> > > >
>> > > > > I think this is basically what we've been proposing for a while
>> now.
>> > > > >
>> > > > >
>> > > > > On Thu, Sep 26, 2013 at 8:29 PM, Michal Mocny <
>> mmocny@chromium.org>
>> > > > wrote:
>> > > > >
>> > > > > > I would suggest perhaps a simpler approach, which doesn't
add
>> > > anything
>> > > > > new
>> > > > > > to cordova-cli/plugman:
>> > > > > >
>> > > > > > - Each plugin ships with a "tests" js-module, and we document
a
>> > > > > convention
>> > > > > > of where they should live, and what signature it should
have
>> (i.e.,
>> > > > > > cordova.require('plugin.name.Tests').forEach(...) ).
>> > > > > >   - Will need a common way to describe/report results (others
>> have
>> > > > > > mentioned TAP).
>> > > > > > - Any app is free to run those plugin tests in any which
way,
>> but
>> > we
>> > > > > ship a
>> > > > > > mobile-spec app which is one opinionated way to do so.
>> > > > > >   - It attempts to require the test module for each installed
>> > plugin,
>> > > > > runs
>> > > > > > them, and aggregates results.
>> > > > > >   - It could report results to some shared server, allow
>> toggling
>> > of
>> > > > > tests,
>> > > > > > etc, but no plugin should know or care about those features.
>> > > > > >
>> > > > > > Using that as a generic base:
>> > > > > >
>> > > > > > - We ship a "CDVTests" (or whatever) plugin which has a
bunch of
>> > > > library
>> > > > > > code for creating tests, and plugins can use it to register
>> their
>> > > > tests.
>> > > > > > - This makes it easier to register manual tests in a common
>> format
>> > > for
>> > > > > core
>> > > > > > plugins, and prevents code duplication for core auto tests.
>> > > > > > - External plugins can chose to use our testing library,
or not.
>> > > > > >
>> > > > > > -Michal
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Sep 26, 2013 at 10:34 AM, Braden Shepherdson <
>> > > > > braden@chromium.org
>> > > > > > >wrote:
>> > > > > >
>> > > > > > > Here's an off-the-top-of-my-head sketch of how we might
do
>> > Voltron
>> > > > > tests:
>> > > > > > >
>> > > > > > > - Add a tag to plugin.xml that names each test file:
>> > > > > > >     <test type="automatic" src="spec/foo.js" name="Foo
>> Automated"
>> > > />
>> > > > > > >     <test type="manual" src="spec/bar.js" name="Foo
Manual" />
>> > > > > > > - Add a new command, cordova test (maybe prepare-test),
that:
>> > > > > > >     - Ignores the top-level www.
>> > > > > > >     - Instead copies in a basic testing index.html
similar to
>> the
>> > > > > current
>> > > > > > > mobile-spec's
>> > > > > > >     - That index reads a file akin to cordova_plugins.js
>> > > > > > (cordova_tests.js,
>> > > > > > > maybe?) generated by the CLI, containing the info from
the
>> <test>
>> > > > tags.
>> > > > > > >     - It has navigation similar to the current mobile-spec,
>> with
>> > > > > buttons
>> > > > > > > for the automatic and manual sections. Auto has "All"
and then
>> > each
>> > > > > > module,
>> > > > > > > manual just has the list of modules.
>> > > > > > >
>> > > > > > > Thoughts?
>> > > > > > >
>> > > > > > > Braden
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, Sep 26, 2013 at 6:33 AM, Carlos Santana <
>> > > > csantana23@gmail.com
>> > > > > > > >wrote:
>> > > > > > >
>> > > > > > > > I like the idea can we call mobilespec now cordova-voltron
>> and
>> > be
>> > > > DRY
>> > > > > > and
>> > > > > > > > use the tests form the plugins.
>> > > > > > > >
>> > > > > > > > Voltron by itself creates an App that tests only
core, but
>> as
>> > you
>> > > > > > > > use plugman to add plugins to voltron it has more
test
>> cases.
>> > > > > > > >
>> > > > > > > > It would not be a bad idea to enhance plugin.xml
in the
>> future
>> > to
>> > > > > > include
>> > > > > > > > information about testing (i.e. Directory containing
tests
>> > files,
>> > > > > test
>> > > > > > > > command, etc..)
>> > > > > > > >
>> > > > > > > > --Carlos
>> > > > > > > >
>> > > > > > > > On Thursday, September 26, 2013, Anis KADRI wrote:
>> > > > > > > >
>> > > > > > > > > What's the challenge of having us use the
tests that come
>> > with
>> > > > the
>> > > > > > > > > individual plugins ?
>> > > > > > > > >
>> > > > > > > > > On Thu, Sep 26, 2013 at 8:13 AM, David Kemp
<
>> > drkemp@google.com
>> > > > > > > > <javascript:;>>
>> > > > > > > > > wrote:
>> > > > > > > > > > Currently, the automated test system
that we have
>> running
>> > > > > (derived
>> > > > > > > from
>> > > > > > > > > > Medic) uses only the mobilespec tests.
It does not yet
>> use
>> > > > tests
>> > > > > > > > > collected
>> > > > > > > > > > from the plugins. Its been talked about,
but not gone
>> > > anywhere.
>> > > > > > > > > >
>> > > > > > > > > > David Kemp
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Wed, Sep 25, 2013 at 7:58 PM, Jesse
<
>> > > > purplecabbage@gmail.com
>> > > > > > > > <javascript:;>>
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > >> Yeah, I have pushed some changes
to mobile-spec, and
>> when
>> > I
>> > > > did
>> > > > > I
>> > > > > > > also
>> > > > > > > > > >> copied the tests into the plugin
involved.
>> > > > > > > > > >> Until we get the magic test runner
happening, I think
>> we
>> > > just
>> > > > > keep
>> > > > > > > > > >> duplicating.
>> > > > > > > > > >>
>> > > > > > > > > >> @purplecabbage
>> > > > > > > > > >> risingj.com
>> > > > > > > > > >>
>> > > > > > > > > >>
>> > > > > > > > > >> On Wed, Sep 25, 2013 at 4:38 PM,
Steven Gill <
>> > > > > > > stevengill97@gmail.com
>> > > > > > > > <javascript:;>
>> > > > > > > > > >
>> > > > > > > > > >> wrote:
>> > > > > > > > > >>
>> > > > > > > > > >> > We copied over tests into plugins
when we first broke
>> > them
>> > > > > out,
>> > > > > > > but
>> > > > > > > > I
>> > > > > > > > > >> don't
>> > > > > > > > > >> > believe they have been updated.
>> > > > > > > > > >> >
>> > > > > > > > > >> > I would say for now to just
add the tests to mobile
>> > spec,
>> > > > and
>> > > > > > > > > possibly in
>> > > > > > > > > >> > the future we go all voltron
to build mobile spec and
>> > keep
>> > > > > tests
>> > > > > > > > with
>> > > > > > > > > >> their
>> > > > > > > > > >> > corresponding plugins.
>> > > > > > > > > >> >
>> > > > > > > > > >> >
>> > > > > > > > > >> > On Wed, Sep 25, 2013 at 4:22
PM, Joe Bowser <
>> > > > > bowserj@gmail.com
>> > > > > > > > <javascript:;>>
>> > > > > > > > > wrote:
>> > > > > > > > > >> >
>> > > > > > > > > >> > > Hey
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Right now, I'm working
on a weird file issue that
>> > > requires
>> > > > > me
>> > > > > > to
>> > > > > > > > > >> > > update mobile-spec, but
I'm wondering where the
>> tests
>> > > > should
>> > > > > > > live.
>> > > > > > > > > >> > > Should it all keep living
in mobile-spec, or is it
>> > with
>> > > > the
>> > > > > > > > plugins.
>> > > > > > > > > >> > > And if it's with the plugins,
will there be
>> scripts to
>> > > > > > assemble
>> > > > > > > > > >> > > mobile-spec all Voltron
style?
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > This came up earlier,
but I haven't found any fix
>> that
>> > > > > needed
>> > > > > > a
>> > > > > > > > > >> > > mobile-spec test.  (Many
that need native testing,
>> > like
>> > > > > > > recursive
>> > > > > > > > > file
>> > > > > > > > > >> > > copy, etc).  Any thoughts?
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Joe
>> > > > > > > > > >> > >
>> > > > > > > > > >> >
>> > > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Carlos Santana
>> > > > > > > > <csantana23@gmail.com>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message