cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy Barkalov (Akvelon)" <v-dmb...@microsoft.com>
Subject RE: [mobile-spec] consistency and reliability
Date Fri, 31 Oct 2014 20:40:07 GMT
Hi Everyone,

It is true that mobile spec test results for Windows are currently not reliable. But our experience
allows us to understand which automated tests are expected to fail on Windows / WP8 platforms
and which are true fails. 

At the same time, we are fixing plugin automated tests for Windows platform and mark unsupported
feature tests as pending (e.g. https://github.com/apache/cordova-plugin-contacts/pull/49,
 https://github.com/apache/cordova-plugin-file/pull/88). We have the same plans for WP8 platform.

To ensure Cordova plugins quality on Windows, we also make complete run for manual plugin
tests in addition to mobile spec tests.


Regards, Dmitriy

-----Original Message-----
From: Toda, Shingo [mailto:ShingoT@fast.au.fujitsu.com] 
Sent: Friday, October 31, 2014 4:57 AM
To: dev@cordova.apache.org
Cc: Toda, Shingo
Subject: RE: [mobile-spec] consistency and reliability

It's great that this topic has been raised as we have some questions on mobile spec.

We need to test Cordova and core plugins to assure ourselves of the quality so we are running
mobile spec and add tests if there are APIs for which we couldn't find tests in mobile spec
but I'm interested in how you test Cordova on various platforms. 

On Android and iOS, mobile spec runs nicely but on Windows 8 mobile spec there are many crashes
due to unsupported API, alert undefined, file paths etc. I am wondering how you test Cordova
for Windows 8.

If you don't mind, I would like to know if you are using (new/old) mobilespec or other test
suite for testing Windows 8.
Is there any strategy for testing platform-specific APIs which not all platforms support?


Regards,
Shingo


-----Original Message-----
From: Brian LeRoux [mailto:b@brian.io]
Sent: Friday, October 31, 2014 11:09 AM
To: dev@cordova.apache.org
Subject: Re: [mobile-spec] consistency and reliability

Nope.

On Thu, Oct 30, 2014, 5:04 PM Jesse <purplecabbage@gmail.com> wrote:

> I would much rather we fix things, than continually rewrite + discard, 
> which seems to be the norm these days.
> tape/tap would require us to throw away thousands of jasmine2 based 
> tests wouldn't it?
>
>
> @purplecabbage
> risingj.com
>
> On Thu, Oct 30, 2014 at 4:44 PM, Brian LeRoux <b@brian.io> wrote:
>
> > tape (and or tap) has no globals and simple / shallow callback 
> > solution which fixes that
> >
> > we will have fresh intern blood to sacrifice to the test lib gods 
> > alter soon..
> >
> > On Thu, Oct 30, 2014, 4:10 PM Michal Mocny <mmocny@chromium.org> wrote:
> >
> > > I've seen that in the past and it usually happens due to misuse of
> > jasmine,
> > > which seems unfortunately quite brittle.  Specifically, if done() 
> > > isn't called, or an async event fires after a test has already 
> > > failed / timed
> > out
> > > and calls done(), or registers new tests with it() etc, weird 
> > > things
> will
> > > happen.
> > >
> > > Usually its a sign that a test is not handling failures 
> > > appropriately,
> > but
> > > it is a bit unfortunate that jasmine is so brittle.
> > >
> > > -Michal
> > >
> > > On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo <
> > ShingoT@fast.au.fujitsu.com
> > > >
> > > wrote:
> > >
> > > > This is exactly what I got too.
> > > >
> > > > Regards,
> > > > Shingo
> > > >
> > > > From: Shazron [mailto:shazron@gmail.com]
> > > > Sent: Friday, October 31, 2014 9:02 AM
> > > > To: dev@cordova.apache.org
> > > > Cc: Toda, Shingo
> > > > Subject: Re: [mobile-spec] consistency and reliability
> > > >
> > > > Seems to be exactly what I am seeing.
> > > > Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
> > > >
> > > > On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <
> > > ShingoT@fast.au.fujitsu.com
> > > > <mailto:ShingoT@fast.au.fujitsu.com>> wrote:
> > > > Hi Shazron
> > > >
> > > > Our team are running mobilespec for Android, iOS and Windows. I
> cannot
> > > see
> > > > the attachment but we also have been seeing what you’ve gotten 
> > > > since
> > new
> > > > style mobilespec was introduced. It seems that each plugin test 
> > > > run
> > works
> > > > fine but if all tests get run at once I am likely to get more 
> > > > than
> one
> > > > report in a run reporting that there are a number of specs like
> > > > 900
> > specs
> > > > (I don’t think specs are so much).
> > > >
> > > > I noticed that some of the specs were called more than once when 
> > > > I
> got
> > > > multiple reports. I usually disable FileTransfer, Globalization 
> > > > and
> > Media
> > > > specs at first then run test for them individually because I saw 
> > > > some
> > of
> > > > those tests got called more than once. This could be just my 
> > > > case so
> it
> > > > might not be yours.
> > > >
> > > > Regards,
> > > > Shingo
> > > >
> > > > From: Shazron
> > > > [mailto:shazron@gmail.com<mailto:shazron@gmail.com>]
> > > > Sent: Friday, October 31, 2014 7:54 AM
> > > > To: dev@cordova.apache.org<mailto:dev@cordova.apache.org>
> > > > Subject: [mobile-spec] consistency and reliability
> > > >
> > > >
> > > > `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios 
> > > > --globalplugins`
> > > >
> > > > Has anyone gotten mobile-spec to work consistently? I get wildly
> > > different
> > > > results on each run. Sometimes there is more than one report in 
> > > > a
> run.
> > > See:
> > > >
> > > > [Inline image 1]
> > > >
> > > >
> > >
> >
>
Mime
View raw message