Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: pass (asf.osuosl.org: domain of mloenko@gmail.com designates
 66.249.92.206 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=HWv1Ecie/MlsctEvxwsBix45rXsm2UrMys8a5isSMWxU+9/1mjWJKNfOp5J9HbRP/KQh8HZBIm1OKhiiVfjcyQKg/u0Lhew/hAU1ataU99BBequqSng4mKbSi+Qix7LqmBtnIgTh3nWyTLVfC1bLscPxLIa5kMV4ar7gfHu52RM=
Message-ID: <906dd82e0601270533r35ae49b0j7161127cf3a8e551@mail.gmail.com>
Date: Fri, 27 Jan 2006 19:33:35 +0600
From: Mikhail Loenko <mloenko@gmail.com>
To: harmony-dev@incubator.apache.org
Subject: Re: [testing] code for exotic configurations
In-Reply-To: <43DA1A8E.7020207@googlemail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
References: <43D9DF6C.5080508@gmail.com>
	 <OF2772C9EA.40D4EC3D-ON80257103.0036A017-80257103.00395123@uk.ibm.com>
	 <906dd82e0601270441y54061d5k9415277613735dc7@mail.gmail.com>
	 <43DA1A8E.7020207@googlemail.com>

Hi George,

On 1/27/06, George Harley <george.c.harley@googlemail.com> wrote:
> Sure. And it should be the role of the test framework to inform users
> about which tests got run, passed, failed, got skipped etc.

see below

> > It would not disturb most of the people because the test will pass in '=
bad'
> > environment. But those, who know about these tests will sometimes grep
> > logs to validate configuration.
> >
> Why use logs to tell us information that the unit test framework can alre=
ady
> provide ? Who wants to have to grep through log files when the test runne=
r
> can provide us with what we need ?

Sorry but I did not quite understand. I do not see anything related to
the 'skipped' status in the junit.framework.TestResult

If there is such a status then it would be excellent, it is the option #1 i=
n the
first mail in the thread

Thanks,
Mikhail

>
> > Thanks,
> > Mikhail
> >
> >
> >
> >> Alternatively, they could be
> >> included as part of a general test suite but be purposely skipped over=
 at
> >> test execution time using a
> >> test exclusion list understood by the test runner.
> >>
> >>
> >> Best regards,
> >> George
> >> ________________________________________
> >> George C. Harley
> >>
> >>
> >>
> >>
> >>
> >> Tim Ellison <t.p.ellison@gmail.com>
> >> 27/01/2006 08:53
> >> Please respond to
> >> harmony-dev@incubator.apache.org
> >>
> >>
> >> To
> >> harmony-dev@incubator.apache.org
> >> cc
> >>
> >> Subject
> >> Re: [testing] code for exotic configurations
> >>
> >>
> >>
> >>
> >>
> >>
> >> Anton Avtamonov wrote:
> >>
> >>>> Note that I could create my own provider and test with it, but what =
I
> >>>>
> >> would
> >>
> >>>> really want is to test how my EncryptedPrivateKeyInfo works with
> >>>> AlgorithmParameters from real provider as well as how my other class=
es
> >>>>
> >> work
> >>
> >>>> with real implementations of crypto Engines.
> >>>>
> >>>> Thanks,
> >>>> Mikhail.
> >>>>
> >>>>
> >>> Hi Mikhail,
> >>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> >>> developer-level. Each unit test is intended to test just a limited
> >>> piece of functionality separately from other sub-systems (test for on=
e
> >>> fucntion, test for one class, etc). Such tests must create a desired
> >>> environment over the testing fucntionality and run the scenario in th=
e
> >>> predefined conditions. Unit tests usually able to cover all scenarios
> >>> (execution paths) for the tested parts of fucntionality.
> >>>
> >>> What are you talking about looks like 'system' testing. Such tests
> >>> usually run on the real environment and test the most often scenariou=
s
> >>> (the reduntant set, all scenarios usually cannot be covered). Such
> >>> testing is not concentrated on the particular fucntionality, but
> >>> covers the work of the whole system.
> >>> A sample is: "run some demo application on some particular platform,
> >>> with some particular providers installed and perform some operations"=
.
> >>>
> >>> I think currently we should focus on 'unit' test approach since it is
> >>> more applicable during the development (so my advise is to revert you=
r
> >>> tests to install 'test' providers with the desired behavior as George
> >>> proposed).
> >>> However we should think about 'system' scenarios which can be run on
> >>> the later stage and act as 'verification' of proper work of the entir=
e
> >>> system.
> >>>
> >> I agree with all this.  The unit tests are one style of test for
> >> establishing the correctness of the code.  As you point out the unit
> >> tests typically require a well-defined environment in which to run, an=
d
> >> it becomes a judgment-call as to whether a particular test's
> >> environmental requirements are 'reasonable' or not.
> >>
> >> For example, you can reasonably expect all developers to have an
> >> environment to run unit tests that has enough RAM and a writable disk
> >> etc. such that if those things do not exist the tests will simply fail=
.
> >>  However, you may decide it is unreasonable to expect the environment =
to
> >> include a populated LDAP server, or a carefully configured RMI server.
> >> If you were to call that environment unreasonable then testing JNDI an=
d
> >> RMI would likely involve mock objects etc. to get good unit tests.
> >>
> >> Of course, as you point out, once you are passing the unit tests you
> >> also need the 'system' tests to ensure the code works in a real
> >> environment.  Usage scenarios based on the bigger system are good, as =
is
> >> running the bigger system's test suite on our runtime.
> >>
> >> Regards,
> >> Tim
> >>
> >>
> >>
> >>> --
> >>> Anton Avtamonov,
> >>> Intel Middleware Products Division
> >>>
> >>>
> >> --
> >>
> >> Tim Ellison (t.p.ellison@gmail.com)
> >> IBM Java technology centre, UK.
> >>
> >>
> >>
> >>
> >
> >
>
>