cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse <purplecabb...@gmail.com>
Subject Re: Mobile spec tests and exclusion list
Date Mon, 28 Oct 2013 17:40:00 GMT
Nevermind then.  Doesn't really matter considering tests will be broken out
into each repo anyway.


@purplecabbage
risingj.com


On Mon, Oct 28, 2013 at 9:22 AM, Brian LeRoux <b@brian.io> wrote:

> I grok the theory here but not sure I agree w/ the practice. The working
> 'master' should always be green. Working branches should be the place where
> failing tests happen. If I'm looking to collaborate I won't be hacking on
> any branches w/ failing tests b/c I'll just assume the developers are in
> progress or they don't know wtf they are doing.
>
>
> On Mon, Oct 28, 2013 at 8:56 AM, purplecabbage <purplecabbage@gmail.com
> >wrote:
>
> > Seeing an expected failure on their device may be the impetus for
> > Microsoft or any other device maker to change the api.
> > To me, hiding or not executing the test means we surrender to the fact
> > that it will never pass. Also, there could be cases where a solution
> exists
> > but is not yet known, and seeing the failure could push someone to
> discover
> > it.
> >
> > Sent from my iPhone
> >
> > > On Oct 28, 2013, at 8:19 AM, David Kemp <drkemp@google.com> wrote:
> > >
> > > Specifically I am thinking of  a test that passes on one
> platform/device
> > > but will not pass on another one.
> > > so maybe 'take them out' was poor language.
> > >
> > > If at all possible, the test should be coded in some way to skip it or
> > > change the expectation for that platform/device, rather than 'just
> > knowing'
> > > that test 46 always fails on platform xx.
> > >
> > > This could be done in the test itself (inspect cordova-device or
> > something)
> > > or by some external file that describes expected results.
> > > I would generally rather see it done explicitly in the test.
> > >
> > >
> > >
> > >
> > >> On Mon, Oct 28, 2013 at 10:10 AM, Michal Mocny <mmocny@chromium.org>
> > wrote:
> > >>
> > >> Some test frameworks just have an "expectFailure", so a failed test
> > >> actually lets the test suite pass, and a passed test makes it fail.
> > >>
> > >> -Michal
> > >>
> > >>
> > >>> On Mon, Oct 28, 2013 at 8:17 AM, David Kemp <drkemp@google.com>
> wrote:
> > >>>
> > >>> -1 for known failing tests. You need to have them all pass for a
> clean
> > >> run.
> > >>> If the tests don't work, take them out.
> > >>>
> > >>> I would support some additional functionality to the test runner to
> > allow
> > >>> marking tests.
> > >>> We definitely have tests that are know to not work on a platform, OS
> > >>> version or device.
> > >>> Being able to embody that info in the test system would be great.
> > >>>
> > >>> Until we get more stuff cleaned up we also have tests that are flakey
> > and
> > >>> probably should just trigger a rerun if they fail.
> > >>> My preference is to just fix those though.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Sat, Oct 26, 2013 at 11:02 PM, purplecabbage <
> > purplecabbage@gmail.com
> > >>>> wrote:
> > >>>
> > >>>> Having a known failure in the tests on wp7 is no biggie, it has
> always
> > >>>> been there. Just move on ...
> > >>>>
> > >>>> Sent from my iPhone
> > >>>>
> > >>>>> On Oct 26, 2013, at 3:24 PM, Michal Mocny <mmocny@chromium.org>
> > >> wrote:
> > >>>>>
> > >>>>> We have a proposal and prototype on the table right now for
> > >> re-working
> > >>>>> tests to ship with plugins, defined according to auto and manual
> > >> tests.
> > >>>>>
> > >>>>> To accomplish what you ask for would require a specialized
testing
> > >> app
> > >>>> that
> > >>>>> simply runs both at the same time. (this wouldn't be the default,
> but
> > >>>> would
> > >>>>> be easy to make).
> > >>>>>
> > >>>>> Thus, I think the tests shouldn't be modified (its hard to
state at
> > >>> test
> > >>>>> definition time in which fashion they should be used), the
test
> > >> runner
> > >>>>> should.  This wont solve the problem today, but perhaps in
about a
> > >>> month
> > >>>> it
> > >>>>> could.
> > >>>>>
> > >>>>> -Micha
> > >>>>>
> > >>>>>
> > >>>>> On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) <
> > >>>>> v-segreb@microsoft.com> wrote:
> > >>>>>
> > >>>>>> Hi Michael,
> > >>>>>>
> > >>>>>> Agree. But taking into account having a way to run all
the tests
> > >>>>>> (including ones w/ user interaction) is very useful for
Windows
> > >> Phone
> > >>> I
> > >>>>>> propose the following
> > >>>>>> 1. No changes for non-WP platforms
> > >>>>>> 2. For WP
> > >>>>>> a) Use the following condition for the tests which require
user
> > >>>>>> interaction
> > >>>>>>   define(..., function(...) {
> > >>>>>>     if (isWP8 && !runAll) return;
> > >>>>>>     expect(...);
> > >>>>>>     ...
> > >>>>>>   })
> > >>>>>> b) Current autotests will run w/o runAll option so won't
require
> > >> user
> > >>>>>> interaction
> > >>>>>> c) Add 'Run All Tests (Extended)' option specifically for
WP8
> where
> > >>> we
> > >>>>>> will have runAll == true
> > >>>>>>
> > >>>>>> Motivation:
> > >>>>>> 1. I don't think we should move such tests to manual tests
for WP
> > >> only
> > >>>> to
> > >>>>>> be consistent with other platforms - we actually test api
call and
> > >>> check
> > >>>>>> result
> > >>>>>> 2. By default all tests will run w/o any user interaction
> > >>>>>> 3. We will have an option to quickly check all api before
release
> > >> via
> > >>>> Run
> > >>>>>> All Tests (Extended). In other case we should have special
> > >> information
> > >>>> how
> > >>>>>> to check all the api and not to forget to run such special
tests.
> > >>>>>>
> > >>>>>> Thx!
> > >>>>>> Sergey
> > >>>>>> -----Original Message-----
> > >>>>>> From: mmocny@google.com [mailto:mmocny@google.com] On Behalf
Of
> > >>> Michal
> > >>>>>> Mocny
> > >>>>>> Sent: Saturday, October 26, 2013 4:12 AM
> > >>>>>> To: dev
> > >>>>>> Subject: Re: Mobile spec tests and exclusion list
> > >>>>>>
> > >>>>>> Auto tests should run automatically without intervention.
 If user
> > >>>> actions
> > >>>>>> is needed for test to pass, we should call that something
> different
> > >>>> (manual
> > >>>>>> tests have been used).
> > >>>>>>
> > >>>>>> I think some varient of #3 is fine, this isn't a common
problem.
>  I
> > >>>>>> wouldn't even test for Medic specifically, since I want
my auto
> > >> tests
> > >>> to
> > >>>>>> run automatically even when testing by hand.
> > >>>>>>
> > >>>>>> define(..., function(...) {
> > >>>>>> if (isWP8) return;
> > >>>>>> expect(...);
> > >>>>>> ...
> > >>>>>> })
> > >>>>>>
> > >>>>>> -Michal
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon)
<
> > >>>>>> v-segreb@microsoft.com> wrote:
> > >>>>>>
> > >>>>>>> Mobile spec autotests include tests which on some platforms
> require
> > >>>>>>> user interaction to complete. For example, contact
save api on
> > >>> Windows
> > >>>>>>> Phone requires user to manually click on save  button.
This
> > >> prevents
> > >>>>>>> the tests to be run as part of Medic test automation
since test
> > >> app
> > >>>>>>> just hangs on such api calls.
> > >>>>>>>
> > >>>>>>> Is Windows Phone special or there are similar problem
on other
> > >>>> platforms?
> > >>>>>>>
> > >>>>>>> I'm thinking about the following possible approaches:
> > >>>>>>> #1 Ad-hoc solution to Medic - replacing some test files
as part
> of
> > >>>>>>> Medic functionality (some additional wp specific build
step).
> > >>>>>>> #2 Extending mobile spec functionality- adding something
like
> tests
> > >>>>>>> exclusion config where you can define test ids (or
even the whole
> > >>> api)
> > >>>>>>> to be skipped. Such exclusion list could be generated
on the fly
> > >> and
> > >>>>>>> put to the app before starting tests.
> > >>>>>>> #3 If there are only few such tests we can probably
add check for
> > >> the
> > >>>>>>> current platform to determine whether to include the
test. For
> > >>> example:
> > >>>>>>> if(!(window.MedicTestRunner && isWP8)) {testDefinition}
 Or the
> > >> same
> > >>>>>>> way but inside the test to fail gracefully.
> > >>>>>>>
> > >>>>>>> Thoughts?
> > >>>>>>>
> > >>>>>>> Thx!
> > >>>>>>> Sergey
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message