cordova-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian LeRoux...@brian.io>
Subject Re: Understanding the intent of the test suites and how they fit together
Date Thu, 04 Jul 2013 17:11:53 GMT
More comments inline.


>> * Specifically, the plugin coverage is not tested much in this suite. I
>> am guessing it is assumed that plugins are covered by
>> cordova-mobile-spec?
>>
>
> Correct! Even more: In 3.0, what plugin tests that do exists here will be
> removed.

To expand on this further, indeed it is correct but coverage is never
a goal we want to promote. (Or, at least traditionally we did not.)
Coverage is a poor metric. It leads crummy tests that do not test code
intent and often just slows a test suite down. This leads to tests
becoming a ceremony rather than a practice.

We take testing very seriously and we don't write tests for their own
sake. In fact, I'd prefer we only wrote tests for our own dev flow and
reported defects. This was the philosophy of the original committers
and I recognize things might be different now w/ the project diversity
that we enjoy. I'll happily discuss in further detail if there is
disagreement w/ any of these thoughts.



>> * Seems strange to not expect 100% success - it risks real bugs falling
>> through cracks if everybody assumes fail is normal. Apparently this is a
>> known issue on dev list.
>>
> true :(

It would seem strange if you are new to mobile testing. Many phones do
not share features. For example, a Nexus 7 has one camera and not two.
Failure with a broad API such as our 2.x series is expected behavior.
In our 3.x series with plugins 'broken out' this problem should be
localized to plugins themselves.



>> * Also includes benchmark test cases (I haven't looked at these - they
>> are not part of the auto-tests - any documentation on them?)
>>
> These are pretty much still a work in progress. They are not run unless
> someone is working on performance.

We really need a champion here. Would LOVE to see this get automated.



>> * Most JUnit test cases seem focussed on behaviour of the
>> CordovaWebView. It looks like lots of Java code does not get touched by
>> test cases at all.
>>
> Yep, we'd like this suite to be far more extensive.

Indeed!


>> * There seems no explicit tests for any plugins. Omitted deliberately
>> assuming cordova-mobile-spec will work through all this code, or to do?
>>
> Omitted deliberately I think.

What plugins do you mean?



>> * There seems no explicit test for the Java-side of the message bridge.
>> To do?
>>
> There are bridge tests in the mobile-spec that test them.

We have tried to keep as much testing infra in the JS side for two reasons:

1. The webview is the abstraction we deliver to our developer
community. This is less true in the case of custom plugins but still a
vast majority use case.

2. The JS is cross platform and native tests can get out of sync. Is
the Objective C test correct or the Java one? Nobody would know.



>> * Manual tests suffer lots of fails in my environment - so much so that
>> it's not even clear if these manual tests are expected to work at all,
>> or are they just a bunch of ad-hoc test fragments which were useful once
>> but have since fallen on hard times
>>
>
> The intention is that all manual ones are run for each release, and that
> they should pass.

Yes.

Mime
View raw message