harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley1 <GHAR...@uk.ibm.com>
Subject Re: [testing] code for exotic configurations
Date Fri, 27 Jan 2006 10:26:00 GMT
Hi, 


> I agree with all this.  The unit tests are one style of test for
> establishing the correctness of the code.  As you point out the unit
> tests typically require a well-defined environment in which to run, and
> it becomes a judgment-call as to whether a particular test's
> environmental requirements are 'reasonable' or not.
> 

Yes, and that's something that needs to be carefully weighed up by the 
developer before committing any such unit test code into the test suite. 
If the environment required to run a test case cannot be easily recreated
by other developers then that case should not be checked in.


> For example, you can reasonably expect all developers to have an
> environment to run unit tests that has enough RAM and a writable disk
> etc. such that if those things do not exist the tests will simply fail.
> However, you may decide it is unreasonable to expect the environment to
> include a populated LDAP server, or a carefully configured RMI server.
> If you were to call that environment unreasonable then testing JNDI and
> RMI would likely involve mock objects etc. to get good unit tests.

Right, and it should be the responsibility of the test creator to ensure 
that those mocks and stubs (and
whatever other artifacts are needed) are checked in as part of the test. 
It should not be left to other
developers to spend a lot of time setting up the required environment for 
a test. Tests that are a pain 
to configure will just end up not being run.

But because we live in a less than ideal world there will, no doubt, be 
some tests that will demand an 
environment that is impossible or at the very least difficult to mock up 
for the majority of developers/testers. 
One solution could be to segregate those tests into a separate test suite 
(available for all but primarily
for those working in the niche area that demands the special environment). 
Alternatively, they could be
included as part of a general test suite but be purposely skipped over at 
test execution time using a 
test exclusion list understood by the test runner. 


Best regards, 
George
________________________________________
George C. Harley





Tim Ellison <t.p.ellison@gmail.com> 
27/01/2006 08:53
Please respond to
harmony-dev@incubator.apache.org


To
harmony-dev@incubator.apache.org
cc

Subject
Re: [testing] code for exotic configurations






Anton Avtamonov wrote:
>> Note that I could create my own provider and test with it, but what I 
would
>> really want is to test how my EncryptedPrivateKeyInfo works with
>> AlgorithmParameters from real provider as well as how my other classes 
work
>> with real implementations of crypto Engines.
>>
>> Thanks,
>> Mikhail.
>>
> 
> Hi Mikhail,
> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> developer-level. Each unit test is intended to test just a limited
> piece of functionality separately from other sub-systems (test for one
> fucntion, test for one class, etc). Such tests must create a desired
> environment over the testing fucntionality and run the scenario in the
> predefined conditions. Unit tests usually able to cover all scenarios
> (execution paths) for the tested parts of fucntionality.
> 
> What are you talking about looks like 'system' testing. Such tests
> usually run on the real environment and test the most often scenarious
> (the reduntant set, all scenarios usually cannot be covered). Such
> testing is not concentrated on the particular fucntionality, but
> covers the work of the whole system.
> A sample is: "run some demo application on some particular platform,
> with some particular providers installed and perform some operations".
> 
> I think currently we should focus on 'unit' test approach since it is
> more applicable during the development (so my advise is to revert your
> tests to install 'test' providers with the desired behavior as George
> proposed).
> However we should think about 'system' scenarios which can be run on
> the later stage and act as 'verification' of proper work of the entire
> system.

I agree with all this.  The unit tests are one style of test for
establishing the correctness of the code.  As you point out the unit
tests typically require a well-defined environment in which to run, and
it becomes a judgment-call as to whether a particular test's
environmental requirements are 'reasonable' or not.

For example, you can reasonably expect all developers to have an
environment to run unit tests that has enough RAM and a writable disk
etc. such that if those things do not exist the tests will simply fail.
 However, you may decide it is unreasonable to expect the environment to
include a populated LDAP server, or a carefully configured RMI server.
If you were to call that environment unreasonable then testing JNDI and
RMI would likely involve mock objects etc. to get good unit tests.

Of course, as you point out, once you are passing the unit tests you
also need the 'system' tests to ensure the code works in a real
environment.  Usage scenarios based on the bigger system are good, as is
running the bigger system's test suite on our runtime.

Regards,
Tim


> --
> Anton Avtamonov,
> Intel Middleware Products Division
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.



Mime
View raw message