incubator-callback-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon MacDonald <simon.macdon...@gmail.com>
Subject Re: tagging 1.5rc friday
Date Tue, 28 Feb 2012 15:16:32 GMT
Well that process is a lot like what I do but there is one proto step I do
before anything else. First I run the mobile-spec qunit tests with the
currently released version. I make note of how many tests are passing and
failing (as well as which tests fail). I use this as a baseline for step 1.
If more tests fail with the proposed releases we've taken a step backwards.

Once upon a time I started working on a doc that explicitly stated the
results expected of each manual test but I've never been able to finish it.

Qunit is a good way to test to make sure all the objects of the API exist
but what I believe Becky was referring to and I agree that it does not give
us 100% coverage. For instance, the qunit tests can check for a Media
object but they can't tell you if the sound really plays or it can check
on the video capture API but it can't really grab a video as it requires
user interaction.

Simon Mac Donald
http://hi.im/simonmacdonald


On Mon, Feb 27, 2012 at 4:50 PM, Filip Maj <fil@adobe.com> wrote:

> Testing process: this is important :)
>
> Let's hash that out in this thread, and document over at
> http://wiki.apache.org/cordova/RunningTests
>
> Here's my version, please add anything that you think I'm missing. For me
> it's as simple as:
>
> 1. Run the qunit tests in mobile-spec. This is a baseline. I have never
> seen it pass 100% but below 50 tests failing (less than 10%) seems to be
> what the average across platforms is. Our goal is to reduce this as much
> as possible.
> 2. Run the "manual" tests in mobile-spec. These are the individual
> links/pages showing up when you run mobile-spec.
>
> I would like to see the manual tests refined a bit more to have expected
> behaviour mapped out more obviously. Something like:
>  - you run the accelerometer manual tests.
>  - these tests ask you to lie the device flat on a table and hit a
> button. We assert that the various x/y/z accel values line up to what we
> expect.
>  - do the same for various other device positions.
>  - rinse and repeat for all of our APIs.
>
> I had Becky express concerns that mobile-spec is not a complete test suite
> and "needs improvement." Please, if you feel this is the case, it is
> *extremely* important to commit patches to the test suite! It is our only
> tool that allows us to say with confidence "we are ok to release."
>
> Moving forward, with cordova-js lined up as dropping in the next few
> releases, as well as the embeddable webview + rename changes coming into
> the native side on all of our platform implementations, the test suite
> needs to be our go-to method for determining what works and what doesn't.
> If you feel it is lacking: patch it.
>
> Let's get this process nailed down in our wiki and in this thread so our
> next release goes smoother.
>
> On 12-02-27 1:35 PM, "Simon MacDonald" <simon.macdonald@gmail.com> wrote:
>
> >Well running mobile spec is only part of the regression I usually do.
> >Typically I'll run the autotests with the previous version then again with
> >the latest version and find the differences. By doing that this release I
> >was able to fix the issue with the DirectoryReader on Android. Then I'll
> >run through the manual tests in mobile spec to make sure that things are
> >still working. Then I start going through the examples on
> >docs.phonegap.combecause it is really embarrassing when the sample
> >code in our docs doesn't
> >run.
> >
> >I've been doing this for a number of releases now and it helps me get a
> >feel for how stable we are. With all the moving parts for 1.5 I don't feel
> >like we are as stable as 1.4 and don't think we should tag tomorrow as
> >there are some more things I want to look at.
> >
> >Simon Mac Donald
> >http://hi.im/simonmacdonald
> >
> >
> >On Mon, Feb 27, 2012 at 4:30 PM, Joe Bowser <bowserj@gmail.com> wrote:
> >
> >> What regression process are you using?  I've personally have never seen
> >> mobile-spec pass on Android, so I'm curious as to what the criteria is.
> >>
> >> Joe
> >>
> >> On Mon, Feb 27, 2012 at 1:24 PM, Simon MacDonald
> >> <simon.macdonald@gmail.com>wrote:
> >>
> >> > I do not feel comfortable tagging 1.5.0 tomorrow. I'm still finding
> >>bugs
> >> in
> >> > regression.
> >> >
> >> > Simon Mac Donald
> >> > http://hi.im/simonmacdonald
> >> >
> >> >
> >> > On Mon, Feb 27, 2012 at 4:20 PM, Joe Bowser <bowserj@gmail.com>
> wrote:
> >> >
> >> > > JIRA says that this is supposed to happen tomorrow.
> >> > >
> >> > > On Mon, Feb 27, 2012 at 1:18 PM, Simon MacDonald
> >> > > <simon.macdonald@gmail.com>wrote:
> >> > >
> >> > > > Wait, when are we supposed to be releasing 1.5?
> >> > > >
> >> > > > Simon Mac Donald
> >> > > > http://hi.im/simonmacdonald
> >> > > >
> >> > > >
> >> > > > On Mon, Feb 27, 2012 at 4:10 PM, Steven Gill
> >><stevengill97@gmail.com
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > I haven't pulled the release candidate together yet, so
feel
> >>free
> >> to
> >> > > > retag.
> >> > > > >
> >> > > > > -Steve
> >> > > > >
> >> > > > > On Mon, Feb 27, 2012 at 12:58 PM, Becky Gibson <
> >> > gibson.becky@gmail.com
> >> > > > > >wrote:
> >> > > > >
> >> > > > > > Has anything been done with the build?  I would like
to add
> >>the
> >> > > > > > <type>Entry.toURL() api and deprecate to.URI()
into the
> >>release
> >> > > > > candidate.
> >> > > > > >  Is ready to go, just need to retag if this change
is to go
> >>in.
> >> > > > > >
> >> > > > > > -becky
> >> > > > > >
> >> > > > > >
> >> > > > > > On Mon, Feb 27, 2012 at 12:35 PM, Becky Gibson <
> >> > > gibson.becky@gmail.com
> >> > > > > > >wrote:
> >> > > > > >
> >> > > > > > > iOS is tagged for 1.5.0rc1
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Sat, Feb 25, 2012 at 12:31 AM, Simon MacDonald
<
> >> > > > > > > simon.macdonald@gmail.com> wrote:
> >> > > > > > >
> >> > > > > > >> One test case was using escapeURL when it
should have used
> >> > > escapeURI
> >> > > > > so
> >> > > > > > I
> >> > > > > > >> fixed it and checked it in. Also, in common-js
there was a
> >>bug
> >> > in
> >> > > > the
> >> > > > > > >> readEntries method where it was referencing
this.fullPath
> >>when
> >> > it
> >> > > > > should
> >> > > > > > >> have been this.path. I fixed that as well.
> >> > > > > > >>
> >> > > > > > >> I've regenerated the cordova.android.js file
and checked
> >>it in
> >> > > along
> >> > > > > > with
> >> > > > > > >> a
> >> > > > > > >> version bump for Android to 1.5.0rc1.
> >> > > > > > >>
> >> > > > > > >> Simon Mac Donald
> >> > > > > > >> http://hi.im/simonmacdonald
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >> On Fri, Feb 24, 2012 at 6:53 PM, Joe Bowser
<
> >> bowserj@gmail.com>
> >> > > > > wrote:
> >> > > > > > >>
> >> > > > > > >> > Which test case?
> >> > > > > > >> >
> >> > > > > > >> > On Fri, Feb 24, 2012 at 12:23 PM, Simon
MacDonald <
> >> > > > > > >> > simon.macdonald@gmail.com
> >> > > > > > >> > > wrote:
> >> > > > > > >> >
> >> > > > > > >> > > I will not be tagging Android today.
There are a few
> >> > > previously
> >> > > > > > >> passing
> >> > > > > > >> > > test cases in mobile-spec that are
now failing. Will
> >>tag
> >> > once
> >> > > > the
> >> > > > > > >> problem
> >> > > > > > >> > > is resolved.
> >> > > > > > >> > >
> >> > > > > > >> > > Simon Mac Donald
> >> > > > > > >> > > http://hi.im/simonmacdonald
> >> > > > > > >> > >
> >> > > > > > >> > >
> >> > > > > > >> > > On Fri, Feb 24, 2012 at 1:54 PM,
Steve Gill <
> >> > > > > stevengill97@gmail.com
> >> > > > > > >
> >> > > > > > >> > > wrote:
> >> > > > > > >> > >
> >> > > > > > >> > > > Hey guys,
> >> > > > > > >> > > >
> >> > > > > > >> > > > Lets tag 1.5.0rc1 today. Post
here once your repo is
> >> > tagged.
> >> > > > > > >> > > >
> >> > > > > > >> > > > -Steve
> >> > > > > > >> > > >
> >> > > > > > >> > > >
> >> > > > > > >> > > >
> >> > > > > > >> > > >
> >> > > > > > >> > > > On Feb 22, 2012, at 9:31 AM,
Brian LeRoux
> >><b@brian.io>
> >> > > wrote:
> >> > > > > > >> > > >
> >> > > > > > >> > > > > notes from call this am
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > - aim to release 1.5 on
tues
> >> > > > > > >> > > > > - cordova js unlikely
for wp, ios, blackberry
> >> > > > > > >> > > > > - cordova rename from
phonegap on track
> >> > > > > > >> > > > > - coho release tool now
does gpg signing: need
> >>input
> >> > from
> >> > > > > > mentors
> >> > > > > > >> on
> >> > > > > > >> > > > > cutting official apache
release
> >> > > > > > >> > > > > - plugins are breaking:
create a wiki page about
> >> > updating
> >> > > > > ---but
> >> > > > > > >> > > > > plugins have bigger problems
which we aim to
> >>address
> >> > > > sometime
> >> > > > > > >> between
> >> > > > > > >> > > > > now and 2.x
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message