cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From purplecabbage <purplecabb...@gmail.com>
Subject Re: Mobile spec tests and exclusion list
Date Mon, 28 Oct 2013 15:56:54 GMT
Seeing an expected failure on their device may be the impetus for Microsoft or any other device
maker to change the api. 
To me, hiding or not executing the test means we surrender to the fact that it will never
pass. Also, there could be cases where a solution exists but is not yet known, and seeing
the failure could push someone to discover it. 

Sent from my iPhone

> On Oct 28, 2013, at 8:19 AM, David Kemp <drkemp@google.com> wrote:
> 
> Specifically I am thinking of  a test that passes on one platform/device
> but will not pass on another one.
> so maybe 'take them out' was poor language.
> 
> If at all possible, the test should be coded in some way to skip it or
> change the expectation for that platform/device, rather than 'just knowing'
> that test 46 always fails on platform xx.
> 
> This could be done in the test itself (inspect cordova-device or something)
> or by some external file that describes expected results.
> I would generally rather see it done explicitly in the test.
> 
> 
> 
> 
>> On Mon, Oct 28, 2013 at 10:10 AM, Michal Mocny <mmocny@chromium.org> wrote:
>> 
>> Some test frameworks just have an "expectFailure", so a failed test
>> actually lets the test suite pass, and a passed test makes it fail.
>> 
>> -Michal
>> 
>> 
>>> On Mon, Oct 28, 2013 at 8:17 AM, David Kemp <drkemp@google.com> wrote:
>>> 
>>> -1 for known failing tests. You need to have them all pass for a clean
>> run.
>>> If the tests don't work, take them out.
>>> 
>>> I would support some additional functionality to the test runner to allow
>>> marking tests.
>>> We definitely have tests that are know to not work on a platform, OS
>>> version or device.
>>> Being able to embody that info in the test system would be great.
>>> 
>>> Until we get more stuff cleaned up we also have tests that are flakey and
>>> probably should just trigger a rerun if they fail.
>>> My preference is to just fix those though.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Sat, Oct 26, 2013 at 11:02 PM, purplecabbage <purplecabbage@gmail.com
>>>> wrote:
>>> 
>>>> Having a known failure in the tests on wp7 is no biggie, it has always
>>>> been there. Just move on ...
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On Oct 26, 2013, at 3:24 PM, Michal Mocny <mmocny@chromium.org>
>> wrote:
>>>>> 
>>>>> We have a proposal and prototype on the table right now for
>> re-working
>>>>> tests to ship with plugins, defined according to auto and manual
>> tests.
>>>>> 
>>>>> To accomplish what you ask for would require a specialized testing
>> app
>>>> that
>>>>> simply runs both at the same time. (this wouldn't be the default, but
>>>> would
>>>>> be easy to make).
>>>>> 
>>>>> Thus, I think the tests shouldn't be modified (its hard to state at
>>> test
>>>>> definition time in which fashion they should be used), the test
>> runner
>>>>> should.  This wont solve the problem today, but perhaps in about a
>>> month
>>>> it
>>>>> could.
>>>>> 
>>>>> -Micha
>>>>> 
>>>>> 
>>>>> On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) <
>>>>> v-segreb@microsoft.com> wrote:
>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> Agree. But taking into account having a way to run all the tests
>>>>>> (including ones w/ user interaction) is very useful for Windows
>> Phone
>>> I
>>>>>> propose the following
>>>>>> 1. No changes for non-WP platforms
>>>>>> 2. For WP
>>>>>> a) Use the following condition for the tests which require user
>>>>>> interaction
>>>>>>   define(..., function(...) {
>>>>>>     if (isWP8 && !runAll) return;
>>>>>>     expect(...);
>>>>>>     ...
>>>>>>   })
>>>>>> b) Current autotests will run w/o runAll option so won't require
>> user
>>>>>> interaction
>>>>>> c) Add 'Run All Tests (Extended)' option specifically for WP8 where
>>> we
>>>>>> will have runAll == true
>>>>>> 
>>>>>> Motivation:
>>>>>> 1. I don't think we should move such tests to manual tests for WP
>> only
>>>> to
>>>>>> be consistent with other platforms - we actually test api call and
>>> check
>>>>>> result
>>>>>> 2. By default all tests will run w/o any user interaction
>>>>>> 3. We will have an option to quickly check all api before release
>> via
>>>> Run
>>>>>> All Tests (Extended). In other case we should have special
>> information
>>>> how
>>>>>> to check all the api and not to forget to run such special tests.
>>>>>> 
>>>>>> Thx!
>>>>>> Sergey
>>>>>> -----Original Message-----
>>>>>> From: mmocny@google.com [mailto:mmocny@google.com] On Behalf Of
>>> Michal
>>>>>> Mocny
>>>>>> Sent: Saturday, October 26, 2013 4:12 AM
>>>>>> To: dev
>>>>>> Subject: Re: Mobile spec tests and exclusion list
>>>>>> 
>>>>>> Auto tests should run automatically without intervention.  If user
>>>> actions
>>>>>> is needed for test to pass, we should call that something different
>>>> (manual
>>>>>> tests have been used).
>>>>>> 
>>>>>> I think some varient of #3 is fine, this isn't a common problem.
 I
>>>>>> wouldn't even test for Medic specifically, since I want my auto
>> tests
>>> to
>>>>>> run automatically even when testing by hand.
>>>>>> 
>>>>>> define(..., function(...) {
>>>>>> if (isWP8) return;
>>>>>> expect(...);
>>>>>> ...
>>>>>> })
>>>>>> 
>>>>>> -Michal
>>>>>> 
>>>>>> 
>>>>>> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon) <
>>>>>> v-segreb@microsoft.com> wrote:
>>>>>> 
>>>>>>> Mobile spec autotests include tests which on some platforms require
>>>>>>> user interaction to complete. For example, contact save api on
>>> Windows
>>>>>>> Phone requires user to manually click on save  button. This
>> prevents
>>>>>>> the tests to be run as part of Medic test automation since test
>> app
>>>>>>> just hangs on such api calls.
>>>>>>> 
>>>>>>> Is Windows Phone special or there are similar problem on other
>>>> platforms?
>>>>>>> 
>>>>>>> I'm thinking about the following possible approaches:
>>>>>>> #1 Ad-hoc solution to Medic - replacing some test files as part
of
>>>>>>> Medic functionality (some additional wp specific build step).
>>>>>>> #2 Extending mobile spec functionality- adding something like
tests
>>>>>>> exclusion config where you can define test ids (or even the whole
>>> api)
>>>>>>> to be skipped. Such exclusion list could be generated on the
fly
>> and
>>>>>>> put to the app before starting tests.
>>>>>>> #3 If there are only few such tests we can probably add check
for
>> the
>>>>>>> current platform to determine whether to include the test. For
>>> example:
>>>>>>> if(!(window.MedicTestRunner && isWP8)) {testDefinition}
 Or the
>> same
>>>>>>> way but inside the test to fail gracefully.
>>>>>>> 
>>>>>>> Thoughts?
>>>>>>> 
>>>>>>> Thx!
>>>>>>> Sergey
>> 

Mime
View raw message