cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian LeRoux...@brian.io>
Subject Re: Mobile spec tests and exclusion list
Date Mon, 28 Oct 2013 16:22:55 GMT
I grok the theory here but not sure I agree w/ the practice. The working
'master' should always be green. Working branches should be the place where
failing tests happen. If I'm looking to collaborate I won't be hacking on
any branches w/ failing tests b/c I'll just assume the developers are in
progress or they don't know wtf they are doing.


On Mon, Oct 28, 2013 at 8:56 AM, purplecabbage <purplecabbage@gmail.com>wrote:

> Seeing an expected failure on their device may be the impetus for
> Microsoft or any other device maker to change the api.
> To me, hiding or not executing the test means we surrender to the fact
> that it will never pass. Also, there could be cases where a solution exists
> but is not yet known, and seeing the failure could push someone to discover
> it.
>
> Sent from my iPhone
>
> > On Oct 28, 2013, at 8:19 AM, David Kemp <drkemp@google.com> wrote:
> >
> > Specifically I am thinking of  a test that passes on one platform/device
> > but will not pass on another one.
> > so maybe 'take them out' was poor language.
> >
> > If at all possible, the test should be coded in some way to skip it or
> > change the expectation for that platform/device, rather than 'just
> knowing'
> > that test 46 always fails on platform xx.
> >
> > This could be done in the test itself (inspect cordova-device or
> something)
> > or by some external file that describes expected results.
> > I would generally rather see it done explicitly in the test.
> >
> >
> >
> >
> >> On Mon, Oct 28, 2013 at 10:10 AM, Michal Mocny <mmocny@chromium.org>
> wrote:
> >>
> >> Some test frameworks just have an "expectFailure", so a failed test
> >> actually lets the test suite pass, and a passed test makes it fail.
> >>
> >> -Michal
> >>
> >>
> >>> On Mon, Oct 28, 2013 at 8:17 AM, David Kemp <drkemp@google.com> wrote:
> >>>
> >>> -1 for known failing tests. You need to have them all pass for a clean
> >> run.
> >>> If the tests don't work, take them out.
> >>>
> >>> I would support some additional functionality to the test runner to
> allow
> >>> marking tests.
> >>> We definitely have tests that are know to not work on a platform, OS
> >>> version or device.
> >>> Being able to embody that info in the test system would be great.
> >>>
> >>> Until we get more stuff cleaned up we also have tests that are flakey
> and
> >>> probably should just trigger a rerun if they fail.
> >>> My preference is to just fix those though.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Sat, Oct 26, 2013 at 11:02 PM, purplecabbage <
> purplecabbage@gmail.com
> >>>> wrote:
> >>>
> >>>> Having a known failure in the tests on wp7 is no biggie, it has always
> >>>> been there. Just move on ...
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On Oct 26, 2013, at 3:24 PM, Michal Mocny <mmocny@chromium.org>
> >> wrote:
> >>>>>
> >>>>> We have a proposal and prototype on the table right now for
> >> re-working
> >>>>> tests to ship with plugins, defined according to auto and manual
> >> tests.
> >>>>>
> >>>>> To accomplish what you ask for would require a specialized testing
> >> app
> >>>> that
> >>>>> simply runs both at the same time. (this wouldn't be the default,
but
> >>>> would
> >>>>> be easy to make).
> >>>>>
> >>>>> Thus, I think the tests shouldn't be modified (its hard to state
at
> >>> test
> >>>>> definition time in which fashion they should be used), the test
> >> runner
> >>>>> should.  This wont solve the problem today, but perhaps in about
a
> >>> month
> >>>> it
> >>>>> could.
> >>>>>
> >>>>> -Micha
> >>>>>
> >>>>>
> >>>>> On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) <
> >>>>> v-segreb@microsoft.com> wrote:
> >>>>>
> >>>>>> Hi Michael,
> >>>>>>
> >>>>>> Agree. But taking into account having a way to run all the tests
> >>>>>> (including ones w/ user interaction) is very useful for Windows
> >> Phone
> >>> I
> >>>>>> propose the following
> >>>>>> 1. No changes for non-WP platforms
> >>>>>> 2. For WP
> >>>>>> a) Use the following condition for the tests which require user
> >>>>>> interaction
> >>>>>>   define(..., function(...) {
> >>>>>>     if (isWP8 && !runAll) return;
> >>>>>>     expect(...);
> >>>>>>     ...
> >>>>>>   })
> >>>>>> b) Current autotests will run w/o runAll option so won't require
> >> user
> >>>>>> interaction
> >>>>>> c) Add 'Run All Tests (Extended)' option specifically for WP8
where
> >>> we
> >>>>>> will have runAll == true
> >>>>>>
> >>>>>> Motivation:
> >>>>>> 1. I don't think we should move such tests to manual tests for
WP
> >> only
> >>>> to
> >>>>>> be consistent with other platforms - we actually test api call
and
> >>> check
> >>>>>> result
> >>>>>> 2. By default all tests will run w/o any user interaction
> >>>>>> 3. We will have an option to quickly check all api before release
> >> via
> >>>> Run
> >>>>>> All Tests (Extended). In other case we should have special
> >> information
> >>>> how
> >>>>>> to check all the api and not to forget to run such special tests.
> >>>>>>
> >>>>>> Thx!
> >>>>>> Sergey
> >>>>>> -----Original Message-----
> >>>>>> From: mmocny@google.com [mailto:mmocny@google.com] On Behalf
Of
> >>> Michal
> >>>>>> Mocny
> >>>>>> Sent: Saturday, October 26, 2013 4:12 AM
> >>>>>> To: dev
> >>>>>> Subject: Re: Mobile spec tests and exclusion list
> >>>>>>
> >>>>>> Auto tests should run automatically without intervention.  If
user
> >>>> actions
> >>>>>> is needed for test to pass, we should call that something different
> >>>> (manual
> >>>>>> tests have been used).
> >>>>>>
> >>>>>> I think some varient of #3 is fine, this isn't a common problem.
 I
> >>>>>> wouldn't even test for Medic specifically, since I want my auto
> >> tests
> >>> to
> >>>>>> run automatically even when testing by hand.
> >>>>>>
> >>>>>> define(..., function(...) {
> >>>>>> if (isWP8) return;
> >>>>>> expect(...);
> >>>>>> ...
> >>>>>> })
> >>>>>>
> >>>>>> -Michal
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon) <
> >>>>>> v-segreb@microsoft.com> wrote:
> >>>>>>
> >>>>>>> Mobile spec autotests include tests which on some platforms
require
> >>>>>>> user interaction to complete. For example, contact save
api on
> >>> Windows
> >>>>>>> Phone requires user to manually click on save  button. This
> >> prevents
> >>>>>>> the tests to be run as part of Medic test automation since
test
> >> app
> >>>>>>> just hangs on such api calls.
> >>>>>>>
> >>>>>>> Is Windows Phone special or there are similar problem on
other
> >>>> platforms?
> >>>>>>>
> >>>>>>> I'm thinking about the following possible approaches:
> >>>>>>> #1 Ad-hoc solution to Medic - replacing some test files
as part of
> >>>>>>> Medic functionality (some additional wp specific build step).
> >>>>>>> #2 Extending mobile spec functionality- adding something
like tests
> >>>>>>> exclusion config where you can define test ids (or even
the whole
> >>> api)
> >>>>>>> to be skipped. Such exclusion list could be generated on
the fly
> >> and
> >>>>>>> put to the app before starting tests.
> >>>>>>> #3 If there are only few such tests we can probably add
check for
> >> the
> >>>>>>> current platform to determine whether to include the test.
For
> >>> example:
> >>>>>>> if(!(window.MedicTestRunner && isWP8)) {testDefinition}
 Or the
> >> same
> >>>>>>> way but inside the test to fail gracefully.
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>> Thx!
> >>>>>>> Sergey
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message