harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Loenko <mloe...@gmail.com>
Subject Re: [testing] code for exotic configurations
Date Fri, 27 Jan 2006 13:47:08 GMT
On 1/27/06, Geir Magnusson Jr <geir@pobox.com> wrote:
> > If the test can be configured by a few people only who works on that
> > specific area and those people are aware of those tests why not just
> > print a log when the test is skipped?
>
> Because the same set of people that will be bothered by separate suites
> will have the same reaction to skipped tests.

By 'skipped' in this context I meant that the test did not verify the code but
reported the 'passed' status. That set of people will not even notice
these tests.

Thanks,
Mikhail

>
> This is why I advocate making a "separate tree" for the system tests -
> make it clear that they are not the general unit tests...
>
> >
> > It would not disturb most of the people because the test will pass in 'bad'
> > environment. But those, who know about these tests will sometimes grep
> > logs to validate configuration.
>
> IMO, there's too much special information there, too much config.  I'm a
> simple person, and like things clean and simple.  I don't like to mix
> concerns when possible, and here's a place where it's definitely
> possible to separate cleanly.
>
> I don't see the downside.
>
> geir
>
> >
> > Thanks,
> > Mikhail
> >
> >
> >> Alternatively, they could be
> >> included as part of a general test suite but be purposely skipped over at
> >> test execution time using a
> >> test exclusion list understood by the test runner.
> >>
> >>
> >> Best regards,
> >> George
> >> ________________________________________
> >> George C. Harley
> >>
> >>
> >>
> >>
> >>
> >> Tim Ellison <t.p.ellison@gmail.com>
> >> 27/01/2006 08:53
> >> Please respond to
> >> harmony-dev@incubator.apache.org
> >>
> >>
> >> To
> >> harmony-dev@incubator.apache.org
> >> cc
> >>
> >> Subject
> >> Re: [testing] code for exotic configurations
> >>
> >>
> >>
> >>
> >>
> >>
> >> Anton Avtamonov wrote:
> >>>> Note that I could create my own provider and test with it, but what
I
> >> would
> >>>> really want is to test how my EncryptedPrivateKeyInfo works with
> >>>> AlgorithmParameters from real provider as well as how my other classes
> >> work
> >>>> with real implementations of crypto Engines.
> >>>>
> >>>> Thanks,
> >>>> Mikhail.
> >>>>
> >>> Hi Mikhail,
> >>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> >>> developer-level. Each unit test is intended to test just a limited
> >>> piece of functionality separately from other sub-systems (test for one
> >>> fucntion, test for one class, etc). Such tests must create a desired
> >>> environment over the testing fucntionality and run the scenario in the
> >>> predefined conditions. Unit tests usually able to cover all scenarios
> >>> (execution paths) for the tested parts of fucntionality.
> >>>
> >>> What are you talking about looks like 'system' testing. Such tests
> >>> usually run on the real environment and test the most often scenarious
> >>> (the reduntant set, all scenarios usually cannot be covered). Such
> >>> testing is not concentrated on the particular fucntionality, but
> >>> covers the work of the whole system.
> >>> A sample is: "run some demo application on some particular platform,
> >>> with some particular providers installed and perform some operations".
> >>>
> >>> I think currently we should focus on 'unit' test approach since it is
> >>> more applicable during the development (so my advise is to revert your
> >>> tests to install 'test' providers with the desired behavior as George
> >>> proposed).
> >>> However we should think about 'system' scenarios which can be run on
> >>> the later stage and act as 'verification' of proper work of the entire
> >>> system.
> >> I agree with all this.  The unit tests are one style of test for
> >> establishing the correctness of the code.  As you point out the unit
> >> tests typically require a well-defined environment in which to run, and
> >> it becomes a judgment-call as to whether a particular test's
> >> environmental requirements are 'reasonable' or not.
> >>
> >> For example, you can reasonably expect all developers to have an
> >> environment to run unit tests that has enough RAM and a writable disk
> >> etc. such that if those things do not exist the tests will simply fail.
> >>  However, you may decide it is unreasonable to expect the environment to
> >> include a populated LDAP server, or a carefully configured RMI server.
> >> If you were to call that environment unreasonable then testing JNDI and
> >> RMI would likely involve mock objects etc. to get good unit tests.
> >>
> >> Of course, as you point out, once you are passing the unit tests you
> >> also need the 'system' tests to ensure the code works in a real
> >> environment.  Usage scenarios based on the bigger system are good, as is
> >> running the bigger system's test suite on our runtime.
> >>
> >> Regards,
> >> Tim
> >>
> >>
> >>> --
> >>> Anton Avtamonov,
> >>> Intel Middleware Products Division
> >>>
> >> --
> >>
> >> Tim Ellison (t.p.ellison@gmail.com)
> >> IBM Java technology centre, UK.
> >>
> >>
> >>
> >
> >
>

Mime
View raw message