Mailing-List: contact dev-help@cordova.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cordova.apache.org
Received-SPF: pass (athena.apache.org: domain of drkemp@google.com designates
 209.85.212.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <526c822d.caa7420a.6b6e.5b46@mx.google.com>
References: 
 <c2629defed294edab470c7edbc465b3d@BLUPR03MB263.namprd03.prod.outlook.com>
	<CAEeF2Te=AgVcWB4wtjZx3jFmmz+=rjMGoc9ZV1i_owkxCsOQ3A@mail.gmail.com>
	<fa3f1af05f104877b8c6756f0b72db92@BLUPR03MB263.namprd03.prod.outlook.com>
	<CAEeF2TfPRP=wcjXRbtuiwey3knvajpbnKvET8zBRf1B_pdqCYQ@mail.gmail.com>
	<526c822d.caa7420a.6b6e.5b46@mx.google.com>
Date: Mon, 28 Oct 2013 08:17:55 -0400
Message-ID: 
 <CAMTxO41TVLO9QT=qFv3EC+QH1Qjsyv10JrYDnZA4ZmZu62Q6Kg@mail.gmail.com>
Subject: Re: Mobile spec tests and exclusion list
From: David Kemp <drkemp@google.com>
To: dev@cordova.apache.org
Content-Type: multipart/alternative; boundary=047d7b6721c4039f6c04e9cc170a

--047d7b6721c4039f6c04e9cc170a
Content-Type: text/plain; charset=ISO-8859-1

-1 for known failing tests. You need to have them all pass for a clean run.
If the tests don't work, take them out.

I would support some additional functionality to the test runner to allow
marking tests.
We definitely have tests that are know to not work on a platform, OS
version or device.
Being able to embody that info in the test system would be great.

Until we get more stuff cleaned up we also have tests that are flakey and
probably should just trigger a rerun if they fail.
My preference is to just fix those though.


On Sat, Oct 26, 2013 at 11:02 PM, purplecabbage <purplecabbage@gmail.com>wrote:

> Having a known failure in the tests on wp7 is no biggie, it has always
> been there. Just move on ...
>
> Sent from my iPhone
>
> > On Oct 26, 2013, at 3:24 PM, Michal Mocny <mmocny@chromium.org> wrote:
> >
> > We have a proposal and prototype on the table right now for re-working
> > tests to ship with plugins, defined according to auto and manual tests.
> >
> > To accomplish what you ask for would require a specialized testing app
> that
> > simply runs both at the same time. (this wouldn't be the default, but
> would
> > be easy to make).
> >
> > Thus, I think the tests shouldn't be modified (its hard to state at test
> > definition time in which fashion they should be used), the test runner
> > should.  This wont solve the problem today, but perhaps in about a month
> it
> > could.
> >
> > -Micha
> >
> >
> > On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) <
> > v-segreb@microsoft.com> wrote:
> >
> >> Hi Michael,
> >>
> >> Agree. But taking into account having a way to run all the tests
> >> (including ones w/ user interaction) is very useful for Windows Phone I
> >> propose the following
> >> 1. No changes for non-WP platforms
> >> 2. For WP
> >>  a) Use the following condition for the tests which require user
> >> interaction
> >>    define(..., function(...) {
> >>      if (isWP8 && !runAll) return;
> >>      expect(...);
> >>      ...
> >>    })
> >>  b) Current autotests will run w/o runAll option so won't require user
> >> interaction
> >>  c) Add 'Run All Tests (Extended)' option specifically for WP8 where we
> >> will have runAll == true
> >>
> >> Motivation:
> >> 1. I don't think we should move such tests to manual tests for WP only
> to
> >> be consistent with other platforms - we actually test api call and check
> >> result
> >> 2. By default all tests will run w/o any user interaction
> >> 3. We will have an option to quickly check all api before release via
> Run
> >> All Tests (Extended). In other case we should have special information
> how
> >> to check all the api and not to forget to run such special tests.
> >>
> >> Thx!
> >> Sergey
> >> -----Original Message-----
> >> From: mmocny@google.com [mailto:mmocny@google.com] On Behalf Of Michal
> >> Mocny
> >> Sent: Saturday, October 26, 2013 4:12 AM
> >> To: dev
> >> Subject: Re: Mobile spec tests and exclusion list
> >>
> >> Auto tests should run automatically without intervention.  If user
> actions
> >> is needed for test to pass, we should call that something different
> (manual
> >> tests have been used).
> >>
> >> I think some varient of #3 is fine, this isn't a common problem.  I
> >> wouldn't even test for Medic specifically, since I want my auto tests to
> >> run automatically even when testing by hand.
> >>
> >> define(..., function(...) {
> >>  if (isWP8) return;
> >>  expect(...);
> >>  ...
> >> })
> >>
> >> -Michal
> >>
> >>
> >> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon) <
> >> v-segreb@microsoft.com> wrote:
> >>
> >>> Mobile spec autotests include tests which on some platforms require
> >>> user interaction to complete. For example, contact save api on Windows
> >>> Phone requires user to manually click on save  button. This prevents
> >>> the tests to be run as part of Medic test automation since test  app
> >>> just hangs on such api calls.
> >>>
> >>> Is Windows Phone special or there are similar problem on other
> platforms?
> >>>
> >>> I'm thinking about the following possible approaches:
> >>> #1 Ad-hoc solution to Medic - replacing some test files as part of
> >>> Medic functionality (some additional wp specific build step).
> >>> #2 Extending mobile spec functionality- adding something like tests
> >>> exclusion config where you can define test ids (or even the whole api)
> >>> to be skipped. Such exclusion list could be generated on the fly and
> >>> put to the app before starting tests.
> >>> #3 If there are only few such tests we can probably add check for the
> >>> current platform to determine whether to include the test. For example:
> >>> if(!(window.MedicTestRunner && isWP8)) {testDefinition}  Or the same
> >>> way but inside the test to fail gracefully.
> >>>
> >>> Thoughts?
> >>>
> >>> Thx!
> >>> Sergey
> >>
>

--047d7b6721c4039f6c04e9cc170a--