cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Mocny <mmo...@chromium.org>
Subject Re: Mobile spec tests and exclusion list
Date Mon, 28 Oct 2013 14:10:55 GMT
Some test frameworks just have an "expectFailure", so a failed test
actually lets the test suite pass, and a passed test makes it fail.

-Michal


On Mon, Oct 28, 2013 at 8:17 AM, David Kemp <drkemp@google.com> wrote:

> -1 for known failing tests. You need to have them all pass for a clean run.
> If the tests don't work, take them out.
>
> I would support some additional functionality to the test runner to allow
> marking tests.
> We definitely have tests that are know to not work on a platform, OS
> version or device.
> Being able to embody that info in the test system would be great.
>
> Until we get more stuff cleaned up we also have tests that are flakey and
> probably should just trigger a rerun if they fail.
> My preference is to just fix those though.
>
>
>
>
>
> On Sat, Oct 26, 2013 at 11:02 PM, purplecabbage <purplecabbage@gmail.com
> >wrote:
>
> > Having a known failure in the tests on wp7 is no biggie, it has always
> > been there. Just move on ...
> >
> > Sent from my iPhone
> >
> > > On Oct 26, 2013, at 3:24 PM, Michal Mocny <mmocny@chromium.org> wrote:
> > >
> > > We have a proposal and prototype on the table right now for re-working
> > > tests to ship with plugins, defined according to auto and manual tests.
> > >
> > > To accomplish what you ask for would require a specialized testing app
> > that
> > > simply runs both at the same time. (this wouldn't be the default, but
> > would
> > > be easy to make).
> > >
> > > Thus, I think the tests shouldn't be modified (its hard to state at
> test
> > > definition time in which fashion they should be used), the test runner
> > > should.  This wont solve the problem today, but perhaps in about a
> month
> > it
> > > could.
> > >
> > > -Micha
> > >
> > >
> > > On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) <
> > > v-segreb@microsoft.com> wrote:
> > >
> > >> Hi Michael,
> > >>
> > >> Agree. But taking into account having a way to run all the tests
> > >> (including ones w/ user interaction) is very useful for Windows Phone
> I
> > >> propose the following
> > >> 1. No changes for non-WP platforms
> > >> 2. For WP
> > >>  a) Use the following condition for the tests which require user
> > >> interaction
> > >>    define(..., function(...) {
> > >>      if (isWP8 && !runAll) return;
> > >>      expect(...);
> > >>      ...
> > >>    })
> > >>  b) Current autotests will run w/o runAll option so won't require user
> > >> interaction
> > >>  c) Add 'Run All Tests (Extended)' option specifically for WP8 where
> we
> > >> will have runAll == true
> > >>
> > >> Motivation:
> > >> 1. I don't think we should move such tests to manual tests for WP only
> > to
> > >> be consistent with other platforms - we actually test api call and
> check
> > >> result
> > >> 2. By default all tests will run w/o any user interaction
> > >> 3. We will have an option to quickly check all api before release via
> > Run
> > >> All Tests (Extended). In other case we should have special information
> > how
> > >> to check all the api and not to forget to run such special tests.
> > >>
> > >> Thx!
> > >> Sergey
> > >> -----Original Message-----
> > >> From: mmocny@google.com [mailto:mmocny@google.com] On Behalf Of
> Michal
> > >> Mocny
> > >> Sent: Saturday, October 26, 2013 4:12 AM
> > >> To: dev
> > >> Subject: Re: Mobile spec tests and exclusion list
> > >>
> > >> Auto tests should run automatically without intervention.  If user
> > actions
> > >> is needed for test to pass, we should call that something different
> > (manual
> > >> tests have been used).
> > >>
> > >> I think some varient of #3 is fine, this isn't a common problem.  I
> > >> wouldn't even test for Medic specifically, since I want my auto tests
> to
> > >> run automatically even when testing by hand.
> > >>
> > >> define(..., function(...) {
> > >>  if (isWP8) return;
> > >>  expect(...);
> > >>  ...
> > >> })
> > >>
> > >> -Michal
> > >>
> > >>
> > >> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon) <
> > >> v-segreb@microsoft.com> wrote:
> > >>
> > >>> Mobile spec autotests include tests which on some platforms require
> > >>> user interaction to complete. For example, contact save api on
> Windows
> > >>> Phone requires user to manually click on save  button. This prevents
> > >>> the tests to be run as part of Medic test automation since test  app
> > >>> just hangs on such api calls.
> > >>>
> > >>> Is Windows Phone special or there are similar problem on other
> > platforms?
> > >>>
> > >>> I'm thinking about the following possible approaches:
> > >>> #1 Ad-hoc solution to Medic - replacing some test files as part of
> > >>> Medic functionality (some additional wp specific build step).
> > >>> #2 Extending mobile spec functionality- adding something like tests
> > >>> exclusion config where you can define test ids (or even the whole
> api)
> > >>> to be skipped. Such exclusion list could be generated on the fly and
> > >>> put to the app before starting tests.
> > >>> #3 If there are only few such tests we can probably add check for the
> > >>> current platform to determine whether to include the test. For
> example:
> > >>> if(!(window.MedicTestRunner && isWP8)) {testDefinition}  Or
the same
> > >>> way but inside the test to fail gracefully.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> Thx!
> > >>> Sergey
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message