cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From purplecabbage <purplecabb...@gmail.com>
Subject Re: Mobile spec tests and exclusion list
Date Sun, 27 Oct 2013 03:02:04 GMT
Having a known failure in the tests on wp7 is no biggie, it has always been there. Just move
on ...

Sent from my iPhone

> On Oct 26, 2013, at 3:24 PM, Michal Mocny <mmocny@chromium.org> wrote:
> 
> We have a proposal and prototype on the table right now for re-working
> tests to ship with plugins, defined according to auto and manual tests.
> 
> To accomplish what you ask for would require a specialized testing app that
> simply runs both at the same time. (this wouldn't be the default, but would
> be easy to make).
> 
> Thus, I think the tests shouldn't be modified (its hard to state at test
> definition time in which fashion they should be used), the test runner
> should.  This wont solve the problem today, but perhaps in about a month it
> could.
> 
> -Micha
> 
> 
> On Sat, Oct 26, 2013 at 6:41 AM, Sergey Grebnov (Akvelon) <
> v-segreb@microsoft.com> wrote:
> 
>> Hi Michael,
>> 
>> Agree. But taking into account having a way to run all the tests
>> (including ones w/ user interaction) is very useful for Windows Phone I
>> propose the following
>> 1. No changes for non-WP platforms
>> 2. For WP
>>  a) Use the following condition for the tests which require user
>> interaction
>>    define(..., function(...) {
>>      if (isWP8 && !runAll) return;
>>      expect(...);
>>      ...
>>    })
>>  b) Current autotests will run w/o runAll option so won't require user
>> interaction
>>  c) Add 'Run All Tests (Extended)' option specifically for WP8 where we
>> will have runAll == true
>> 
>> Motivation:
>> 1. I don't think we should move such tests to manual tests for WP only to
>> be consistent with other platforms - we actually test api call and check
>> result
>> 2. By default all tests will run w/o any user interaction
>> 3. We will have an option to quickly check all api before release via Run
>> All Tests (Extended). In other case we should have special information how
>> to check all the api and not to forget to run such special tests.
>> 
>> Thx!
>> Sergey
>> -----Original Message-----
>> From: mmocny@google.com [mailto:mmocny@google.com] On Behalf Of Michal
>> Mocny
>> Sent: Saturday, October 26, 2013 4:12 AM
>> To: dev
>> Subject: Re: Mobile spec tests and exclusion list
>> 
>> Auto tests should run automatically without intervention.  If user actions
>> is needed for test to pass, we should call that something different (manual
>> tests have been used).
>> 
>> I think some varient of #3 is fine, this isn't a common problem.  I
>> wouldn't even test for Medic specifically, since I want my auto tests to
>> run automatically even when testing by hand.
>> 
>> define(..., function(...) {
>>  if (isWP8) return;
>>  expect(...);
>>  ...
>> })
>> 
>> -Michal
>> 
>> 
>> On Fri, Oct 25, 2013 at 4:37 PM, Sergey Grebnov (Akvelon) <
>> v-segreb@microsoft.com> wrote:
>> 
>>> Mobile spec autotests include tests which on some platforms require
>>> user interaction to complete. For example, contact save api on Windows
>>> Phone requires user to manually click on save  button. This prevents
>>> the tests to be run as part of Medic test automation since test  app
>>> just hangs on such api calls.
>>> 
>>> Is Windows Phone special or there are similar problem on other platforms?
>>> 
>>> I'm thinking about the following possible approaches:
>>> #1 Ad-hoc solution to Medic - replacing some test files as part of
>>> Medic functionality (some additional wp specific build step).
>>> #2 Extending mobile spec functionality- adding something like tests
>>> exclusion config where you can define test ids (or even the whole api)
>>> to be skipped. Such exclusion list could be generated on the fly and
>>> put to the app before starting tests.
>>> #3 If there are only few such tests we can probably add check for the
>>> current platform to determine whether to include the test. For example:
>>> if(!(window.MedicTestRunner && isWP8)) {testDefinition}  Or the same
>>> way but inside the test to fail gracefully.
>>> 
>>> Thoughts?
>>> 
>>> Thx!
>>> Sergey
>> 

Mime
View raw message