cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Brooks <mich...@michaelbrooks.ca>
Subject Re: collecting mobile-spec test results and generating reports
Date Thu, 10 Jan 2013 21:55:42 GMT
Great discussion point Marcel!

As Fil has mentioned, we've started some of this work and it aligns very
closely with your described solution.

At the moment, Medic is wired up to our device wall, but I see it extending
well beyond that. I would love to see the day where Apache Cordova
committers can `npm install -g cordova-medic` and run medic locally. As
commits are made to the Apache Cordova projects, your local devices will
automatically run mobile-spec against the latest changes and report back to
the CouchDB instance. In other words, a de-centralized continuous
integration service.

Michael

On Thu, Jan 10, 2013 at 9:58 AM, Filip Maj <fil@adobe.com> wrote:

> Hey Marcel, I am working towards that continuous integration vision.
>
> github.com/filmaj/medic
>
> It is a node process that builds the latest commits coming into android,
> iOS + BlackBerry repos, pairs them with mobile-spec, scans for
> connected/nearby devices, and deploys the mobile-spec app to those
> devices. It modifies mobile-spec so that all test results are sent off to
> a couchdb server. There is a "dashboard" script you can run that is a
> little node server that aggregates test results from this couchdb server
> and displays. The dashboard is also live here:
>
> http://cordova.medic.jit.su/
>
>
> Currently the Adobe Vancouver office device wall has this setup running on
> it (actually, not at this EXACT time but hopefully be back online by
> tomorrow :) ). Very recently the gitpubsub service over at Apache went
> live (try running curl http://urd.zones.apache.org:2069/json) which
> broadcasts all git commits going into all apache git repos. I am in the
> process of wiring up medic to that new service to pick out cordova commits
> as-they-happen.
>
> The idea is, at least now, that this is a setup that we can distribute to
> almost anyone and have them submit tests to our centralized server. So, a
> mobile, distributed CI setup, if you will. It's a lot of fun to play with!
>
> The couchdb server at this point is an Adobe EC2 instance. I haven't set
> up any special permissions (I think it's still in Admin Party mode) so if
> we were to publicize that server I'd have to set it up so that, at the
> minimum, no one can delete test results :)
>
> Overall, though, I like your idea of the opt-in submission button. Would
> love to see that in mobile spec. Also I'd love feedback on medic!
>
> Cheers,
> Fil
>
> On 1/10/13 8:30 AM, "Marcel Kinard" <cmarcelk@gmail.com> wrote:
>
> >Here is something I feel is important and I wanted to see what you think.
> >
> >Problem definition: Cordova is a bit unique in that it not only has to
> >deal with running on a large number of permutations of device models and
> >device OS's, but the testing is started manually by a large number of
> >people using devices in their desk drawer, it is very decentralized. My
> >fear is that there are invisible holes and dups in our testing at the
> >end of a release. Thus we are releasing code for environments we didn't
> >test, but perhaps could have tested. And are we really running both the
> >automatic and manual tests in mobile-spec, or just the automatic tests?
> >In a more generalized perspective, we don't have good visibility to what
> >environments on which tests ran, and visibility to test failures as we
> >evaluate release readiness, other than a few comments on the dev mailing
> >list.
> >
> >Proposed solution: mobile-spec is a reasonably good test suite. But the
> >test results never leave the device. So the suggestion is to add a
> >"Submit test results" button throughout mobile-spec (at each place where
> >test results are generated) that would be an opt-in mechanism to get the
> >pass/fail test results off the device. Included in the submission with
> >the test results would be a bit of metadata, such as the device make and
> >model and OS version, Cordova version, etc. The submission would get
> >posted to a centralized db somewhere. A web-based query could be run
> >against the db to list the submitted results, with the intention to
> >understand which environments have been tested for a particular Cordova
> >version, which tests ran, and which tests failed/succeeded.
> >
> >It should be opt-in instead of opt-out because we generally don't want
> >test results from a developer that is working on a new feature in the
> >middle of a dev cycle. The intent here is to capture tests from rc's and
> >similar as we approach the end of a release. It's about evaluating
> >release readiness. Maybe in the long term we want to evaluate release
> >readiness all the way through a release, but that's not the short-term
> >intention here.
> >
> >I love the vision of continuous integration test, and see this proposal
> >as a first step towards that.
> >
> >Start shooting holes: Reply with what you think. Before we dive into
> >where the db should be hosted and a wire protocol for the submission, at
> >a high level do you agree with the problem definition and do you have
> >any other ideas/comments on a solution? And what do you feel is the
> >urgency for this? Thanks!
> >
> >-- Marcel Kinard
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message