Mailing-List: contact dev-help@cordova.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cordova.apache.org
Received-SPF: pass (athena.apache.org: domain of purplecabbage@gmail.com
 designates 209.85.214.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <BLUPR03MB3764BAABD6D9BE5482F3093D1150@BLUPR03MB376.namprd03.prod.outlook.com>
References: 
 <BLUPR03MB37690D8985C393973F438F0D1160@BLUPR03MB376.namprd03.prod.outlook.com>
	<CAE6O_-aCh-tgqejocq-pSsvdN8eWEeLQjMxqRX4aBLmWxyx+bg@mail.gmail.com>
	<12E5EC1C-8E0D-4B13-9B18-ABB1869D038B@microsoft.com>
	<BY2PR03MB5729E54C21EC1A1690A6ACB89170@BY2PR03MB572.namprd03.prod.outlook.com>
	<CAKP6k2vmM_61VLAJef87T6hYv6WEtPd8FBORA09sqLuEMw0RNw@mail.gmail.com>
	<BLUPR03MB37675A253FEBD40388E43ACD1170@BLUPR03MB376.namprd03.prod.outlook.com>
	<CAEeF2TcBAb0x7uh=Gh6fLZ5T63RD7t3LrbzJ8q9NWMF47i4B=A@mail.gmail.com>
	<BLUPR03MB3764BAABD6D9BE5482F3093D1150@BLUPR03MB376.namprd03.prod.outlook.com>
Date: Fri, 27 Feb 2015 14:58:42 -0800
Message-ID: 
 <CAKP6k2vj+R9ygASFb4y1D6af2f-bsrNeUSACM2Ptfk4v+tageA@mail.gmail.com>
Subject: Re: Cordova Automated Testing
From: Jesse <purplecabbage@gmail.com>
To: "dev@cordova.apache.org" <dev@cordova.apache.org>
Content-Type: multipart/alternative; boundary=001a1134e0ba5527f7051019cf17

--001a1134e0ba5527f7051019cf17
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Personally, I never use mobile-spec because it is buggy on windows+wp8, and
nearly impossible to tell if a failure is a bad test or bad code, (this may
be better now, but I've already lost faith).  I needed a way to easily test
a particular plugin in isolation, which is why I wrote paramedic.

The cordova-plugin-test-framework will automagically run the auto-tests if
there is a medic.json file in the app package.  I leveraged this to get the
test results posted back to the command line in paramedic.

Ultimately the intent of paramedic is to run jasmine-2 based tests written
according to cordova-plugin-test-frameworks's idioms for every pull request
to every core plugin and display the results back in github.

I will add the ability to specify multiple plugins as an arg, the rest I am
somewhat indifferent about.
Manual tests will always have a place because many APIs require user
interaction.  mobile-spec is still good at this, flakey UI aside.
Ultimately though, paramedic is designed for non-interactive testing, like
what would be done by TravisCI and hopefully soon, Appveyor.


@purplecabbage
risingj.com

On Fri, Feb 27, 2015 at 2:30 PM, Dmitry Blotsky <dblotsky@microsoft.com>
wrote:

> Thanks for the feedback, Michal! The issue we had with using the
> checkboxes was that it was not easily automatable. A more simple solution
> was to use a parameter to specify the plugins you want installed, instead
> of installing all of them and then disabling them. Also the
> dependencies-plugin in mobilespec is a hardcoded list, which requires mor=
e
> manipulation to modify than specifying a list as a parameter. Using it as
> the default is a great idea, but being able to override it during
> invocation would be great too I think. What do you think?
>
> Also: Jesse, and others, do you think it might be worthwhile to
> collaborate and incorporate the features of paramedic (like specifying th=
e
> plugins to run) into mobilespec? That way we will have one tool for testi=
ng
> throughout. I've definitely been sold on the idea of having mobilespec be
> the main point for running tests. That way we can use it in all our testi=
ng
> scenarios: local, Medic, Travis, etc.
>
> Kindly,
> Dmitry
>
> -----Original Message-----
> From: mmocny@google.com [mailto:mmocny@google.com] On Behalf Of Michal
> Mocny
> Sent: Wednesday, February 25, 2015 12:42 PM
> To: dev
> Subject: Re: Cordova Automated Testing
>
> I haven't followed this full thread, so sorry if this is out of context,
> but wanted to point out:
>
> - Jason added support for testing a subset of plugins in mobile-spec by
> way of checkboxes in the automated test runner.  It defaults to all, but
> you can change that.
> - By design, the cordova-plugin-test-framework (and thus mobile-spec) wil=
l
> only run tests for the plugins you have actually installed (actually, for
> the nested plugin tests you have explicitly installed).  The
> createmobilespec.js script will by default install all core plugins, but
> you can easily change that.
>
> -Michal
>
> On Wed, Feb 25, 2015 at 3:20 PM, Dmitry Blotsky <dblotsky@microsoft.com>
> wrote:
>
> > Thanks for the feedback, Jesse and Dmitriy!
> >
> > Judging from the responses, it definitely sounds like modifying
> > mobilespec, or just plain using paramedic would be the way to go. I'm
> > very much swinging in that direction. Medivac really only exists on
> > its own because I wasn't comfortable enough with our git repos to make
> > a branch off of mobilespec; because it really is just mobilespec with
> > some things stripped away.
> >
> > Jesse: to me the primary value add was actually just simplicity. This
> > guy doesn't depend on plugin-test-framework and has none of the
> > heavier features like local storage and test segmentation. It's
> > basically just a Cordova app with a single page to run Jasmine tests
> > and an optional URI to post the results to, and the value is
> > simplicity. It has very little code and has very few moving parts.
> >
> > Dmitriy Barkalov: I don't see what the use case is for adding
> > plugin-test-framework to any Cordova app other than a blank one, and
> > that is basically just doing manually what mobilespec does automaticall=
y.
> >
> > Kindly,
> > Dmitry
> >
> > -----Original Message-----
> > From: Jesse [mailto:purplecabbage@gmail.com]
> > Sent: Wednesday, February 25, 2015 11:36 AM
> > To: dev@cordova.apache.org
> > Subject: Re: Cordova Automated Testing
> >
> > Thanks for contributing, however, I do not see where the value add is
> > in this.
> > mobile-spec does much of this already, as does
> > cordova-plugin-test-framework.
> >
> > Adding an argument to test an individual plugin is a good idea, I
> > would definitely support adding this to mobile-spec.
> >
> > My own cordova-paramedic is also designed for simplified plugin
> > testing, so you can test an individual plugin, and see the results
> > back on the command line. Primarily this was intended for CI with
> > Travis, so we could see immediately if a plugin pull request breaks a
> test on a platform.
> >
> > We can discuss further here, but I think medivac would be better
> > served as a fork+pull request to cordova-medic
> >
> > gh:purplecabbage/cordova-paramedic
> >
> >
> > @purplecabbage
> > risingj.com
> >
> > On Wed, Feb 25, 2015 at 9:35 AM, Dmitriy Barkalov (Akvelon) <
> > v-dmbark@microsoft.com> wrote:
> >
> > > Dmitry,
> > >
> > > I like your idea of adding tests from different formats. I.e. you
> > > search for standard Cordova plugin tests like in core plugins and
> > > for non-standard tests like plane test.js file. I think it is a good
> > > point to provide a way for plugin authors to either reformat their
> > > tests to existing format or provide new test format parser.
> > > Unfortunately, we can't run away from requirement to use Jasmin 2.0
> > > but still it is more
> > freedom.
> > >
> > > It also seems that in many ways your application is doing the same
> > > things as plugin testing framework. It provides a page to run tests,
> > > holds Jasmine and puts results into CouchDB. But it is less flexible
> > > because cannot be included into other application, while any Cordova
> > > application can install plugin testing framework and get a page to
> > > run
> > manual and automated tests.
> > >
> > > So I vote for porting some of the new features from this tool back
> > > to plugin testing framework and letting developers reuse this
> functionality.
> > >
> > >
> > > Regards, Dmitriy
> > >
> > > -----Original Message-----
> > > From: Dmitry Blotsky [mailto:dblotsky@microsoft.com]
> > > Sent: Wednesday, February 25, 2015 11:47 AM
> > > To: dev@cordova.apache.org
> > > Subject: Re: Cordova Automated Testing
> > >
> > > Hi Shazron,
> > >
> > > Thank you for your feedback! It actually doesn=E2=80=99t work on iOS =
because
> > > it seems like the most recent whitelist changes broke things. I ran
> > > "npm
> > test"
> > > for the code in master and some of them fail because of whitelisting.
> > > That=E2=80=99s just my intuition though: I don=E2=80=99t know exactly=
 why it=E2=80=99s not
> > > working yet.
> > >
> > > Regarding reporting though, reporting to localhost is actually just
> > > the default - you can pass a host and port on the CLI and it will
> > > report to a CouchDB server like mobilespec did. There is an
> > > immediate next step in my development efforts to make it not tied to
> > > CouchDB, or to at least allow the user to specify the URI format for
> reporting.
> > > Also, thanks for pointing out the =E2=80=98npm install=E2=80=99 requi=
rement. I added
> > > it to the docs. The reporting to console also happens, but it=E2=80=
=99s
> > currently just limited to console.log.
> > > Medivac uses all the standard Jasmine reporters (HTML, console, and
> > JSAPI).
> > >
> > > And indeed, I will definitely drop a line to Jesse. Is it
> > > well-received in the Apache community to schedule meetings to
> > > discuss details, or should detailed discussions also be happening on
> > > the mailing
> > list?
> > >
> > > Kindly,
> > > Dmitry
> > >
> > > > On Feb 24, 2015, at 3:32 PM, Shazron <shazron@gmail.com> wrote:
> > > >
> > > > Also, the tool does not work on the device (I tried iOS), unlike
> > > > mobile-spec since it's trying to connect to localhost for reporting=
.
> > > >
> > > > On Mon, Feb 23, 2015 at 4:30 PM, Dmitry Blotsky
> > > > <dblotsky@microsoft.com>
> > > wrote:
> > > >> Hi list,
> > > >>
> > > >> Over the past few weeks, I've developed an automated
> > > >> mobilespec-like
> > > tool in the process of resuscitating Cordova's CI. It's tentatively
> > > called cordova-medivac, and it can currently be found here:
> > > http://github.com/dblotsky/cordova-medivac. I'd like to pitch it to
> > > the community as a potential alternative or replacement for
> > > mobilespec. There have been several tools in the past few weeks on
> > > the list, and I've tried to incorporate the features they have into
> > > this
> > tool.
> > > >>
> > > >> It contains the following new features:
> > > >>
> > > >> -          Testing any list of plugins at a time - not just the co=
re
> > > ones; they're passed as a CLI argument
> > > >>
> > > >> -          Support for "new-style" plugin-test-framework-only test=
s,
> > as
> > > well just pure Jasmine spec files that don't use test-framework
> > > >>
> > > >> -          No manual interaction required - it is very basic and
> just
> > > runs the Jasmine tests as soon as it starts
> > > >>
> > > >> -          Very little code - it basically just creates a simple
> dummy
> > > app with plugins (like mobilespec)
> > > >>
> > > >> -          Cleaner code - some refactoring was done to remove a fe=
w
> > > code smells
> > > >>
> > > >> -          No flaky UI
> > > >>
> > > >> -          Optionally reporting test results to a URI (currently
> > > assumed to be a CouchDB server)
> > > >>
> > > >> Current shortcomings:
> > > >>
> > > >> -          The special plugins inside cordova-mobile-spec that
> contain
> > > tests have not yet been copied over
> > > >>
> > > >> -          The code is similar, so some functionality is not exact=
ly
> > > the same, so it's not 100% compatible with mobilespec
> > > >>
> > > >> -          Some error-checking is still missing (like verifying th=
at
> > > CLI/JS/lib are locally linked instead of global)
> > > >>
> > > >> -          Cordova-JS Grunt build task is not being automatically
> run
> > > on creation
> > > >>
> > > >> -          Barebones hard-coded list of "core" plugins; doesn't
> > include
> > > convenience lists like the new ones in mobilespec
> > > >>
> > > >> -          Doesn't support current manual tests
> > > >>
> > > >> I'd like to ask you if you would support any of the following
> > > >> proposed
> > > ideas:
> > > >>
> > > >> -          Reach 100% parity and replace cordova-mobile-spec with
> this
> > > tool or,
> > > >>
> > > >> -          If not replace, then port some of the code cleanup and
> new
> > > features from  this tool back to mobilespec or,
> > > >>
> > > >> -          Use this tool for the CI (cordova-medic) only, and keep
> > > using mobilespec for other tasks
> > > >>
> > > >> Kindly,
> > > >> Dmitry
> > > >>
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
> > > > For additional commands, e-mail: dev-help@cordova.apache.org
> > > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
> > > For additional commands, e-mail: dev-help@cordova.apache.org
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
> > > For additional commands, e-mail: dev-help@cordova.apache.org
> > >
> >
>

--001a1134e0ba5527f7051019cf17--