cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filip Maj <...@adobe.com>
Subject Re: collecting mobile-spec test results and generating reports
Date Thu, 10 Jan 2013 17:58:45 GMT
Hey Marcel, I am working towards that continuous integration vision.

github.com/filmaj/medic

It is a node process that builds the latest commits coming into android,
iOS + BlackBerry repos, pairs them with mobile-spec, scans for
connected/nearby devices, and deploys the mobile-spec app to those
devices. It modifies mobile-spec so that all test results are sent off to
a couchdb server. There is a "dashboard" script you can run that is a
little node server that aggregates test results from this couchdb server
and displays. The dashboard is also live here:

http://cordova.medic.jit.su/


Currently the Adobe Vancouver office device wall has this setup running on
it (actually, not at this EXACT time but hopefully be back online by
tomorrow :) ). Very recently the gitpubsub service over at Apache went
live (try running curl http://urd.zones.apache.org:2069/json) which
broadcasts all git commits going into all apache git repos. I am in the
process of wiring up medic to that new service to pick out cordova commits
as-they-happen.

The idea is, at least now, that this is a setup that we can distribute to
almost anyone and have them submit tests to our centralized server. So, a
mobile, distributed CI setup, if you will. It's a lot of fun to play with!

The couchdb server at this point is an Adobe EC2 instance. I haven't set
up any special permissions (I think it's still in Admin Party mode) so if
we were to publicize that server I'd have to set it up so that, at the
minimum, no one can delete test results :)

Overall, though, I like your idea of the opt-in submission button. Would
love to see that in mobile spec. Also I'd love feedback on medic!

Cheers,
Fil

On 1/10/13 8:30 AM, "Marcel Kinard" <cmarcelk@gmail.com> wrote:

>Here is something I feel is important and I wanted to see what you think.
>
>Problem definition: Cordova is a bit unique in that it not only has to
>deal with running on a large number of permutations of device models and
>device OS's, but the testing is started manually by a large number of
>people using devices in their desk drawer, it is very decentralized. My
>fear is that there are invisible holes and dups in our testing at the
>end of a release. Thus we are releasing code for environments we didn't
>test, but perhaps could have tested. And are we really running both the
>automatic and manual tests in mobile-spec, or just the automatic tests?
>In a more generalized perspective, we don't have good visibility to what
>environments on which tests ran, and visibility to test failures as we
>evaluate release readiness, other than a few comments on the dev mailing
>list.
>
>Proposed solution: mobile-spec is a reasonably good test suite. But the
>test results never leave the device. So the suggestion is to add a
>"Submit test results" button throughout mobile-spec (at each place where
>test results are generated) that would be an opt-in mechanism to get the
>pass/fail test results off the device. Included in the submission with
>the test results would be a bit of metadata, such as the device make and
>model and OS version, Cordova version, etc. The submission would get
>posted to a centralized db somewhere. A web-based query could be run
>against the db to list the submitted results, with the intention to
>understand which environments have been tested for a particular Cordova
>version, which tests ran, and which tests failed/succeeded.
>
>It should be opt-in instead of opt-out because we generally don't want
>test results from a developer that is working on a new feature in the
>middle of a dev cycle. The intent here is to capture tests from rc's and
>similar as we approach the end of a release. It's about evaluating
>release readiness. Maybe in the long term we want to evaluate release
>readiness all the way through a release, but that's not the short-term
>intention here.
>
>I love the vision of continuous integration test, and see this proposal
>as a first step towards that.
>
>Start shooting holes: Reply with what you think. Before we dive into
>where the db should be hosted and a wire protocol for the submission, at
>a high level do you agree with the problem definition and do you have
>any other ideas/comments on a solution? And what do you feel is the
>urgency for this? Thanks!
>
>-- Marcel Kinard


Mime
View raw message