Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: pass (asf.osuosl.org: domain of alexei.zakharov@gmail.com
 designates 64.233.166.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=D50KxcAsC0q79pAwFhnU0XeeopMKOhMIGUGpsyYaA44hMYDXFKUQ5XdHNi/8AptBBjYYAw5t/byHEnfOWiX049RsdePGpExitL47ISBxZxdijIV5TEI4BhN/fnVGTJPH6j9gZRfeYAWwMAQCPeKqvU+u/06XpZHtPPI2TXVDZzg=
Message-ID: <2c9597b90606270111w2b12c27eud89e7f128ff70611@mail.gmail.com>
Date: Tue, 27 Jun 2006 12:11:27 +0400
From: "Alexei Zakharov" <alexei.zakharov@gmail.com>
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib][testing] excluding the failed tests
In-Reply-To: <44A0BE6B.9080202@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=WINDOWS-1252; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
References: <7273946b0606260502s3b8ef51bvda10815463c5547d@mail.gmail.com>
	 <906dd82e0606260545p9438c7q8917081235ccf934@mail.gmail.com>
	 <449FE80F.5040305@gmail.com>
	 <7273946b0606262118q38ebeb06k72eb5ac5c69f3270@mail.gmail.com>
	 <44A0BE6B.9080202@gmail.com>

Hi,
+1 for (3), but I think it will be better to define suite() method and
enumerate passing tests there rather than to comment out the code.

2006/6/27, Richard Liang <richard.liangyx@gmail.com>:
> Hello Vladimir,
>
> +1 to option 3) . We shall comment the failed test cases out and add
> FIXME to remind us to diagnose the problems later. ;-)
>
> Vladimir Ivanov wrote:
> > I see your point.
> > But I feel that we can miss regression in non-tested code if we exclude
> > TestCases.
> > Now, for example we miss testing of java.lang.Class/Process/Thread/Stri=
ng
> > and some other classes.
> >
> > While we have failing tests and don't want to pay attention to these
> > failures we can:
> > 1) Leave things as is =96 do not run TestCases with failing tests.
> > 2) Split passing/failing TestCase into separate "failing TestCase" and
> > "passing TestCase" and exclude "failing TestCases". When test or
> > implementation is fixed we move tests from failing TestCase to passing
> > TestCase.
> > 3) Comment failing tests in TestCases. It is better to run 58 tests
> > instead
> > of 0 for String.
> > 4) Run all TestCases, then, compare test run results with the 'list of
> > known
> > failures' and see whether new failures appeared. This, I think, is bett=
er
> > then 1, 2 and 3, but, overhead is that we support 2 lists - list of kno=
wn
> > failing tests and exclude list where we put crashing tests.
> >
> > Thanks, Vladimir
> > On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> >>
> >> Mikhail Loenko wrote:
> >> > Hi Vladimir,
> >> >
> >> > IMHO the tests are to verify that an update does not introduce any
> >> > regression. So there are two options: remember which exactly tests m=
ay
> >> fail
> >> > and remember that all tests must pass. I believe the latter one is
> >> a bit
> >> > easier and safer.
> >>
> >> +1
> >>
> >> Tim
> >>
> >> > Thanks,
> >> > Mikhail
> >> >
> >> > 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
> >> >> Hi,
> >> >> Working with tests I noticed that we are excluding some tests just
> >> >> because
> >> >> several tests from single TestCase fail.
> >> >>
> >> >> For example, the TestCase 'tests.api.java.lang.StringTest' has 60
> >> >> tests and
> >> >> only 2 of them fails. But the build excludes the whole TestCase
> >> and we
> >> >> just
> >> >> miss testing of java.lang.String implementation.
> >> >>
> >> >> Do we really need to exclude TestCases in 'ant test' target?
> >> >>
> >> >> My suggestion is: do not exclude any tests until it crashes VM.
> >> >> If somebody needs a list of tests that always passed a separated
> >> >> target can
> >> >> be added to build.
> >> >>
> >> >> Do you think we should add target 'test-all' to the build?
> >> >>  Thanks, Vladimir
> >> >>
> >> >>
> >> >
> >> > --------------------------------------------------------------------=
-
> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> > For additional commands, e-mail: harmony-dev-help@incubator.apache.o=
rg
> >> >
> >> >
> >>
> >> --
> >>
> >> Tim Ellison (t.p.ellison@gmail.com)
> >> IBM Java technology centre, UK.
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >
>
> --
> Richard Liang
> China Software Development Lab, IBM


--=20
Alexei Zakharov,
Intel Middleware Product Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org