cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filip Maj <maj....@gmail.com>
Subject Re: An idea for manually testing plugins
Date Thu, 08 Dec 2016 00:28:46 GMT
As Alex Sorokin kindly pointed out to me on Slack, we have Appium UI
automation already existing for the contacts [1] and camera [2]
plugins. Our CI system also executes on this to automate the UI
end-to-end and guard against further errors [3]. The CI tosses this
job over to Sauce Labs to do the heavy lifting: Sauce runs the test
[4] from the example from [3], and, based on the logs, looks to
execute all that fun web + native context UI automation -
unfortunately Sauce only stores assets around for a month, so we can't
see the video or screenshots (it's been over a month since the last
contacts PR).

But all that's to say that, with that in place, I think it's mostly a
matter of applying this approach to other plugins. I think the value
of the UI automation-based tests is wayyyy higher than the autotests;
asserting that a JavaScript function exists inside the Cordova webview
is a good sanity check, but asserting that you can pick a contact via
a native UI and have the relevant contact details show up back in the
Cordova webview, in my opinion, is a much more thorough test of the
functions of the plugin APIs.

Lastly, I think documenting how to set up UI automation tests such as
the ones in the camera and contacts plugins in the
cordova-plugin-test-framework readme makes sense.

Let me know what y'all think.

[1] https://github.com/apache/cordova-plugin-contacts/blob/master/appium-tests
[2] https://github.com/apache/cordova-plugin-camera/blob/master/appium-tests
[3] http://cordova-ci.cloudapp.net:8080/view/Pull%20requests/job/cordova-plugin-contacts-pr/lastSuccessfulBuild/PLATFORM=ios/consoleText
[4] https://saucelabs.com/beta/tests/b014505bd8894ff58b383471bdaf4759

On Wed, Dec 7, 2016 at 6:14 AM, Filip Maj <maj.fil@gmail.com> wrote:
> I am planning on putting a proof of concept together for a few plugins, and
> having something like mobile spec be able to string multiple plugins' UI
> automation tests together and assert pass/failure (via its defineAutoTests /
> defineManualTests-like functions).
>
> I'll post more when I have something to show :)
>
> The idea would then be to help guide how to do this kind of testing via the
> framework Cordova has built up for plugin testing via the
> cordova-plugin-test-framework plugin.
>
> On Dec 7, 2016 05:56, "julio cesar sanchez" <jcesarmobile@gmail.com> wrote:
>>
>> I was the other way around, I got Appium working easily, but I was never
>> able to make calabash work in iOS because it always failed to instrument
>> my
>> app.
>> Both worked fine for Android though.
>>
>>
>> 2016-12-07 12:48 GMT+01:00 Trevor Brindle <tabrindle@gmail.com>:
>>
>> > As little market share as windows has, it's hard to say how much effort
>> > should be dedicated to additional testing for it. That said, if windows
>> > testing is deemed necessary, I don't think we have much choice.
>> >
>> > Just keep in mind the new windows support and new iOS driver are all
>> > appium
>> > 1.6, only released less than 60 days ago.
>> >
>> > Perhaps a PoC project should be done for picking the appropriate
>> > technology?
>> >
>> > We are midway through a similar PoC at my place of work, I could
>> > volunteer
>> > some time.
>> >
>> > On Tue, Dec 6, 2016 at 10:02 PM Filip Maj <maj.fil@gmail.com> wrote:
>> >
>> > > Full disclosure: I contribute(d) to appium and worked for Sauce Labs,
>> > >
>> > > so I am pretty biased ;)
>> > >
>> > >
>> > >
>> > > Calabash, unfortunately, is iOS and Android only and does not support
>> > >
>> > > Windows app, whereas Appium does via WinAppDriver [1].
>> > >
>> > >
>> > >
>> > > I agree that appium, at least earlier on, was difficult to set up.
>> > >
>> > > These days it's a simple `npm install`. I think appium has a much more
>> > >
>> > > prolific committership to boot (MSFT contributes to it, for example).
>> > >
>> > > The github network stats for each project back that up as well.
>> > >
>> > >
>> > >
>> > > [1] https://github.com/microsoft/winappdriver
>> > >
>> > >
>> > >
>> > > On Tue, Dec 6, 2016 at 4:02 PM, Trevor Brindle <tabrindle@gmail.com>
>> > > wrote:
>> > >
>> > > > In my experience with appium, it is troublesome to set up and
>> > > > maintain.
>> > > We
>> > >
>> > > > may investigate other frameworks that have similar functionality.
I
>> > have
>> > >
>> > > > had great luck with calabash.
>> > >
>> > > >
>> > >
>> > > >
>> > >
>> > > > On Tue, Dec 6, 2016 at 5:30 PM julio cesar sanchez <
>> > > jcesarmobile@gmail.com>
>> > >
>> > > > wrote:
>> > >
>> > > >
>> > >
>> > > >> +1 to appium
>> > >
>> > > >>
>> > >
>> > > >>
>> > >
>> > > >>
>> > >
>> > > >> 2016-12-05 23:33 GMT+01:00 Filip Maj <maj.fil@gmail.com>:
>> > >
>> > > >>
>> > >
>> > > >>
>> > >
>> > > >>
>> > >
>> > > >> > Hi, it's me again!
>> > >
>> > > >>
>> > >
>> > > >> >
>> > >
>> > > >>
>> > >
>> > > >> > How I'd like to contribute to Cordova is to help automate
the
>> > > >> > stuff
>> > >
>> > > >>
>> > >
>> > > >> > that saves committers having to take the manual time to do
>> > themselves.
>> > >
>> > > >>
>> > >
>> > > >> > I think a good first goal would be to help automate as much
of
>> > >
>> > > >>
>> > >
>> > > >> > platform release testing as possible [1]. The autotests seem
to
>> > > >> > be
>> > >
>> > > >>
>> > >
>> > > >> > handled relatively well by the CI system so far (if anyone
feels
>> > this
>> > >
>> > > >>
>> > >
>> > > >> > is not true, please speak up! I want to hear about what works
and
>> > what
>> > >
>> > > >>
>> > >
>> > > >> > does not). The other part of mobile-spec-based platform release
>> > >
>> > > >>
>> > >
>> > > >> > testing is to "run the manual tests". After reading the
>> > >
>> > > >>
>> > >
>> > > >> > cordova-plugin-test-framework README [2], I thought this
would be
>> > > >> > a
>> > >
>> > > >>
>> > >
>> > > >> > good place to give plugin developers some better tools to
deal
>> > > >> > with
>> > >
>> > > >>
>> > >
>> > > >> > testing complex UI interactions that the autotests can't
handle
>> > > >> > on
>> > >
>> > > >>
>> > >
>> > > >> > their own. I was thinking appium [3] would be a good tool
to
>> > >
>> > > >>
>> > >
>> > > >> > complement that in this case. It gives us UI hooks into both
web
>> > > >> > and
>> > >
>> > > >>
>> > >
>> > > >> > natives contexts within hybrid applications, plus it also
allows
>> > > >> > us
>> > to
>> > >
>> > > >>
>> > >
>> > > >> > inject JavaScript into the web context. Wondering what others
>> > > >> > think?
>> > >
>> > > >>
>> > >
>> > > >> >
>> > >
>> > > >>
>> > >
>> > > >> > I could then foresee, with a little bit of scaffolding, a
way to
>> > >
>> > > >>
>> > >
>> > > >> > string plugins' appium tests together to fully automate the
>> > > >> > 'manual'
>> > >
>> > > >>
>> > >
>> > > >> > testing of plugin tests during platform release testing.
>> > >
>> > > >>
>> > >
>> > > >> >
>> > >
>> > > >>
>> > >
>> > > >> > Let me know what y'all think!
>> > >
>> > > >>
>> > >
>> > > >> >
>> > >
>> > > >>
>> > >
>> > > >> > [1] https://github.com/apache/cordova-coho/blob/master/docs/
>> > >
>> > > >>
>> > >
>> > > >> > platforms-release-process.md#what-to-test
>> > >
>> > > >>
>> > >
>> > > >> > [2] https://github.com/apache/cordova-plugin-test-framework#
>> > >
>> > > >>
>> > >
>> > > >> > defining-manual-tests
>> > >
>> > > >>
>> > >
>> > > >> > [3] appium.io
>> > >
>> > > >>
>> > >
>> > > >> >
>> > >
>> > > >>
>> > >
>> > > >> > ------------------------------------------------------------
>> > ---------
>> > >
>> > > >>
>> > >
>> > > >> > To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
>> > >
>> > > >>
>> > >
>> > > >> > For additional commands, e-mail: dev-help@cordova.apache.org
>> > >
>> > > >>
>> > >
>> > > >> >
>> > >
>> > > >>
>> > >
>> > > >> >
>> > >
>> > > >>
>> > >
>> > > >>
>> > >
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > >
>> > > To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
>> > >
>> > > For additional commands, e-mail: dev-help@cordova.apache.org
>> > >
>> > >
>> > >
>> > >
>> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
For additional commands, e-mail: dev-help@cordova.apache.org


Mime
View raw message