Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: neutral (asf.osuosl.org: 64.74.244.71 is neither permitted nor
 denied by domain of geir@pobox.com)
Message-ID: <43DA18A9.2000609@pobox.com>
Date: Fri, 27 Jan 2006 07:57:13 -0500
From: Geir Magnusson Jr <geir@pobox.com>
Reply-To: geir@pobox.com
User-Agent: Thunderbird 1.5 (Windows/20051201)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [testing] code for exotic configurations
References: <43D9DF6C.5080508@gmail.com>
	 <OF2772C9EA.40D4EC3D-ON80257103.0036A017-80257103.00395123@uk.ibm.com>
 <906dd82e0601270441y54061d5k9415277613735dc7@mail.gmail.com>
In-Reply-To: <906dd82e0601270441y54061d5k9415277613735dc7@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


Mikhail Loenko wrote:
> On 1/27/06, George Harley1 <GHARLEY@uk.ibm.com> wrote:
>> But because we live in a less than ideal world there will, no doubt, be
>> some tests that will demand an
>> environment that is impossible or at the very least difficult to mock up
>> for the majority of developers/testers.
> 
> I'm absolutely agree that we are neither living in the ideal world nor trying
> to make it ideal :)
> 
> So until we got a 'system' test suite why should we weaken existing tests?
> 
>> One solution could be to segregate those tests into a separate test suite
>> (available for all but primarily
>> for those working in the niche area that demands the special environment).
> 
> Moving this kind of tests would affect many people: they will see
> separate suites,
> try, ask questions...
> 
> If the test can be configured by a few people only who works on that
> specific area and those people are aware of those tests why not just
> print a log when the test is skipped?

Because the same set of people that will be bothered by separate suites 
will have the same reaction to skipped tests.

This is why I advocate making a "separate tree" for the system tests - 
make it clear that they are not the general unit tests...

> 
> It would not disturb most of the people because the test will pass in 'bad'
> environment. But those, who know about these tests will sometimes grep
> logs to validate configuration.

IMO, there's too much special information there, too much config.  I'm a 
simple person, and like things clean and simple.  I don't like to mix 
concerns when possible, and here's a place where it's definitely 
possible to separate cleanly.

I don't see the downside.

geir

> 
> Thanks,
> Mikhail
> 
> 
>> Alternatively, they could be
>> included as part of a general test suite but be purposely skipped over at
>> test execution time using a
>> test exclusion list understood by the test runner.
>>
>>
>> Best regards,
>> George
>> ________________________________________
>> George C. Harley
>>
>>
>>
>>
>>
>> Tim Ellison <t.p.ellison@gmail.com>
>> 27/01/2006 08:53
>> Please respond to
>> harmony-dev@incubator.apache.org
>>
>>
>> To
>> harmony-dev@incubator.apache.org
>> cc
>>
>> Subject
>> Re: [testing] code for exotic configurations
>>
>>
>>
>>
>>
>>
>> Anton Avtamonov wrote:
>>>> Note that I could create my own provider and test with it, but what I
>> would
>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>> AlgorithmParameters from real provider as well as how my other classes
>> work
>>>> with real implementations of crypto Engines.
>>>>
>>>> Thanks,
>>>> Mikhail.
>>>>
>>> Hi Mikhail,
>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>> developer-level. Each unit test is intended to test just a limited
>>> piece of functionality separately from other sub-systems (test for one
>>> fucntion, test for one class, etc). Such tests must create a desired
>>> environment over the testing fucntionality and run the scenario in the
>>> predefined conditions. Unit tests usually able to cover all scenarios
>>> (execution paths) for the tested parts of fucntionality.
>>>
>>> What are you talking about looks like 'system' testing. Such tests
>>> usually run on the real environment and test the most often scenarious
>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>> testing is not concentrated on the particular fucntionality, but
>>> covers the work of the whole system.
>>> A sample is: "run some demo application on some particular platform,
>>> with some particular providers installed and perform some operations".
>>>
>>> I think currently we should focus on 'unit' test approach since it is
>>> more applicable during the development (so my advise is to revert your
>>> tests to install 'test' providers with the desired behavior as George
>>> proposed).
>>> However we should think about 'system' scenarios which can be run on
>>> the later stage and act as 'verification' of proper work of the entire
>>> system.
>> I agree with all this.  The unit tests are one style of test for
>> establishing the correctness of the code.  As you point out the unit
>> tests typically require a well-defined environment in which to run, and
>> it becomes a judgment-call as to whether a particular test's
>> environmental requirements are 'reasonable' or not.
>>
>> For example, you can reasonably expect all developers to have an
>> environment to run unit tests that has enough RAM and a writable disk
>> etc. such that if those things do not exist the tests will simply fail.
>>  However, you may decide it is unreasonable to expect the environment to
>> include a populated LDAP server, or a carefully configured RMI server.
>> If you were to call that environment unreasonable then testing JNDI and
>> RMI would likely involve mock objects etc. to get good unit tests.
>>
>> Of course, as you point out, once you are passing the unit tests you
>> also need the 'system' tests to ensure the code works in a real
>> environment.  Usage scenarios based on the bigger system are good, as is
>> running the bigger system's test suite on our runtime.
>>
>> Regards,
>> Tim
>>
>>
>>> --
>>> Anton Avtamonov,
>>> Intel Middleware Products Division
>>>
>> --
>>
>> Tim Ellison (t.p.ellison@gmail.com)
>> IBM Java technology centre, UK.
>>
>>
>>
> 
>