cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filip Maj <maj....@gmail.com>
Subject Re: [DISCUSS] Improving core plugins automated test coverage
Date Fri, 29 Jan 2016 18:05:46 GMT
O hey!

This is pretty sweet, Alexander. It's been so long since I've looked
at medic code, oh my. I feel so old all of a sudden :D

A few notes:

 - The way you use appium in here is mostly for navigating the UI -
that is awesome. Totally agree with this approach - automate all the
things so y'all can get closer and closer to a continuous integration
nirvana.
 - If you can get an appium run to work locally for you, moving to
Sauce Labs is a breeze - you just change the endpoint of the appium
server from wherever you are running it to ondemand.saucelabs.com:80.
Sauce provides "Open Sauce" for open source projects [1] - which
Cordova certainly falls under. That'll give y'all 5 concurrent VMs to
run tests on. Only iOS and Android are supported right now, but I
think it'd be a big win for Cordova to move to this. You don't have to
worry about managing the emulators/simulators, and parallelizing some
of these functional tests helps speed them up, too, giving you faster
feedback. I'm happy to help getting y'all set up here, too.
 - appium, at this time, doesn't expose a command directly to change
the battery level. However, we've filed an issue [2] and hopefully
that will bubble up to the main project in a release or two. So,
functional tests for the battery functionality are probably some of
the more out-of-reach-ones. That said, if you want to use Sauce, and
if we can write the test such that we isolate _just_ the battery
functionality, we can leverage pre-run executables [3] to basically
have an open shell prompt to the VM running the emulator - and `adb`
within it. So we can:
    - take those `adb shell set battery blah` and stuff 'em into a
shell script we can host somewhere
    - specify the URL to the script as a pre-run executable as part of
the appium test's desired capabilities
    - kick off the appium test, which spins up a VM on Sauce's side
    - the VM starts the emulator, then runs your pre-run exec scripts,
full of adb commands to tweak the emulator
    - finally, the appium test automation commands interacting with
the emulator and the cordova app (presumably with mobile-spec in it?)
start executing, and you get test results back

I think we should start with the simplest APIs possible that are easy
to automate. Build out the framework to do that UI testing - and i
highly recommend we leverage the 5 free VMs from sauce to offload that
- and start integrating that into your day-to-day development work.
Once we've got the basics covered, I'm happy to help work through the
more complex testing cases involving disparate device APIs and see
what we can do on the appium (and/or Sauce) side to help make a
fully-automated testing suite a reality for the cordova project.

Let's do this!

-fil

[1] https://saucelabs.com/opensauce/
[2] https://github.com/appium/appium-adb/issues/133
[3] https://wiki.saucelabs.com/display/DOCS/Test+Configuration+Options#TestConfigurationOptions-Pre-runExecutables

On Fri, Jan 29, 2016 at 4:53 AM, Carlos Santana <csantana23@gmail.com> wrote:
> Thanks for the effort into looking into it.
> For now continue as plan, maybe in the future a portion of the tests can be
> ran on Soucelabs if it make sense to complement on what we have with our
> devices and emulators.
> On Fri, Jan 29, 2016 at 3:20 AM Alexander Sorokin (Akvelon) <
> v-alsoro@microsoft.com> wrote:
>
>> It looks like the Saucelabs infrastructure won't suit our needs. A lot of
>> Appium tests for Android platform now need to connect to the emulator
>> directly via adb or telnet to send mock commands.
>>
>> For example, to mock charger in the battery-status tests, we execute "adb
>> shell dumpsys battery set ac 0".
>>
>> It looks like saucelabs won't let us do that, so we're bound to our own
>> devices and emulators.
>>
>> -----Original Message-----
>> From: Alexander Sorokin (Akvelon) [mailto:v-alsoro@microsoft.com]
>> Sent: Friday, January 22, 2016 12:11 PM
>> To: dev@cordova.apache.org
>> Subject: RE: [DISCUSS] Improving core plugins automated test coverage
>>
>> Thanks for suggesting saucelabs! I'll take a look at it.
>>
>> -----Original Message-----
>> From: Carlos Santana [mailto:csantana23@gmail.com]
>> Sent: Thursday, January 21, 2016 4:25 AM
>> To: dev@cordova.apache.org
>> Subject: Re: [DISCUSS] Improving core plugins automated test coverage
>>
>> Wow this is great! not just for our cordova code, but app developers that
>> want to learn how to use appium to test their own plugins and app code
>>
>> Is this something we can run on saucelabs?, I think is free for open
>> source projects.
>>
>>
>> On Wed, Jan 20, 2016 at 8:16 PM Steven Gill <stevengill97@gmail.com>
>> wrote:
>>
>> > Sweet! Thanks for doing this Alexander! Always thought it would be a
>> > great idea to write some our tests with appium
>> >
>> > On Wed, Jan 20, 2016 at 5:07 PM, Joe Bowser <bowserj@gmail.com> wrote:
>> >
>> > > So, Appium works now and I can close the issue complaining that
>> > > Appium doesn't work?
>> > >
>> > > On Wed, Jan 20, 2016, 9:12 AM Alexander Sorokin (Akvelon) <
>> > > v-alsoro@microsoft.com> wrote:
>> > >
>> > > > Hi guys.
>> > > >
>> > > > Lately I've been working on improving automated test coverage of
>> > > > core plugins. Work is still in progress but there are some results
>> > > > that I
>> > want
>> > > > to share.
>> > > >
>> > > > The idea is to use Appium framework to automate core plugins
>> > > > manual
>> > tests
>> > > > in mobilespec app. I've implemented a new command for Medic. It
>> > > > starts
>> > an
>> > > > Appium server instance, looks for the tests, then runs the
>> > > > mobilespec
>> > app
>> > > > from the specified folder and runs the tests.
>> > > >
>> > > > Appium supports only iOS and Android platforms so there are tests
>> > > > only
>> > > for
>> > > > these two, but we can extend the coverage of other platforms using
>> > > > different ui testing frameworks.
>> > > >
>> > > > The tests themselves are written for jasmine-node and are using "wd"
>> > > > module to connect to Appium server. They are currently located in
>> > plugin
>> > > > repos. Here's some links:
>> > > >
>> > > > Test runner (medic):
>> > > > https://github.com/apache/cordova-medic/pull/70
>> > > >
>> > > > Battery Status tests:
>> > > > https://github.com/apache/cordova-plugin-battery-status/pull/29
>> > > >
>> > > > Camera tests:
>> > > > https://github.com/apache/cordova-plugin-camera/pull/156
>> > > >
>> > > > Contacts tests:
>> > > > https://github.com/apache/cordova-plugin-contacts/pull/101
>> > > >
>> > > > Device Motion tests:
>> > > > https://github.com/apache/cordova-plugin-device-motion/pull/39
>> > > >
>> > > > Tests can run on both real devices and emulators/simulators.
>> > > > While developing the tests, I've been using Android emulator and
>> > > > device with API level 19 (Android 4.4, KitKat) and iOS device
>> > > > (iPad 2) with
>> > iOS
>> > > > 7.1 so the tests can fail on other devices/emulators. I'm going to
>> > > > add
>> > > the
>> > > > support for other versions/devices as well as implement more
>> > > > Appium
>> > tests
>> > > > for other plugins in the nearest future.
>> > > >
>> > > > To run the tests:
>> > > > 1. Clone cordova-medic repo
>> > > > 2. Clone tested plugins repos (appium tests are there, in
>> > > > appium-tests
>> > > > folder)
>> > > > 3. Create mobile spec app with tested plugins:
>> > > >        node
>> > > > cordova-mobile-spec/createmobilespec/createmobilespec.js
>> > > > mobspec-android --android --plugins "cordova-plugin-battery-status
>> > > > cordova-plugin-camera cordova-plugin-contacts
>> > > cordova-plugin-device-motion"
>> > > > 4. Build the mobilespec app:
>> > > >        cd mobspec-android && cordova build 5. Start the
>> > > > emulator/connect the device 6. Run the tests using command:
>> > > > node cordova-medic/medic/medic.js appium --app [mobilespec folder]
>> > > > --platform [platform] --device-name [device/emulator name]
>> > > > --platform-version [platform version] --plugins [list of plugins
>> > > > to
>> > test
>> > > > separated by space] [--device (only if running on a real device)]
>> > --udid
>> > > > [iOS real devices only, device UDID] Every argument except
>> > > > --platform is optional. On iOS, --udid is
>> > mandatory
>> > > > if running on real device.
>> > > >
>> > > > For example, android emulator:
>> > > > node cordova-medic/medic/medic.js appium --app mobspec-android
>> > --platform
>> > > > android --device-name appium --platform-version 19 --plugins
>> > > > "cordova-plugin-device-motion"
>> > > >
>> > > > Android device:
>> > > > node cordova-medic/medic/medic.js appium --platform android --app
>> > > > mobspec-android --plugins "cordova-plugin-contacts" --device
>> > > >
>> > > > iOS simulator:
>> > > > node cordova-medic/medic/medic.js appium --app mobspec-ios
>> > > > --platform
>> > ios
>> > > > --plugins "cordova-plugin-contacts"
>> > > >
>> > > > iOS device:
>> > > > node cordova-medic/medic/medic.js appium --app mobspec-ios
>> > > > --platform
>> > ios
>> > > > --device-name "iPad 2" --udid [UDID here] --plugins
>> > > "cordova-plugin-camera
>> > > > cordova-plugin-contacts"
>> > > >
>> > > > I hope we will be running these tests along with other medic tests
>> > > > that are now running on CI in future:
>> > > > https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fci
>> > > > .cordova.io%2f&data=01%7c01%7cv-alsoro%40microsoft.com%7c5c198a04b
>> > > > bef4dd0eab508d32201c51c%7c72f988bf86f141af91ab2d7cd011db47%7c1&sda
>> > > > ta=YrNTt%2f1zkrRjqr3WVSDF37SDgkCbD2KjjDnaxn47bLU%3d
>> > > >
>> > > > Please take a look and say what you think.
>> > > >
>> > > > ------------------------------------------------------------------
>> > > > --- To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
>> > > > For additional commands, e-mail: dev-help@cordova.apache.org
>> > > >
>> > > >
>> > >
>> >
>>  B KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB
>>   [  X  ܚX K  K[XZ[
>>    ] ][  X  ܚX P  ܙ ݘK \ X  K ܙ B  ܈ Y  ] [ۘ[    [X[     K[XZ[
>>    ] Z [    ܙ ݘK \ X  K ܙ B
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
For additional commands, e-mail: dev-help@cordova.apache.org


Mime
View raw message