Mailing-List: contact river-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: river-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of pdoubleya@gmail.com designates
 209.85.212.47 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=th/bpMgNAi07PHlzUeVj8juMtOmY7wOAcMk84mvuN6/MMpvdvP8v6qdy3FOx52n0rN
         NsHYUzq71vVSs1csy6kBJ/OhaMTY3jfo3OulO11B5zBZM4nmhFs7Q2C8spHrlsYmlnPg
         QEgwSwV2svmkpy1n6falX6zRIAZd4TpdAxiac=
MIME-Version: 1.0
In-Reply-To: <4C74245C.4000007@acm.org>
References: <4C6E0F9C.5090302@acm.org>
	<4C6F587F.6080904@zeus.net.au>
	<4C6F60F7.2020305@acm.org>
	<4C6F6DD0.10407@zeus.net.au>
	<4C6FC021.5080904@acm.org>
	<4C6FC72E.4040907@zeus.net.au>
	<4C7016DB.2070205@acm.org>
	<4C70180D.2060203@acm.org>
	<4C70569F.5070203@zeus.net.au>
	<4C70B2E6.9000801@acm.org>
	<4C719365.6050501@acm.org>
	<4C71B92A.40000@acm.org>
	<4C71B974.3040304@zeus.net.au>
	<4C72F571.7000706@acm.org>
	<4C74245C.4000007@acm.org>
Date: Tue, 24 Aug 2010 22:02:33 +0200
Message-ID: <AANLkTimHzX0gEE7E4J+L+6AmhpwiTaj6GG74BcrRo2U9@mail.gmail.com>
Subject: Re: Ignored tests
From: Patrick Wright <pdoubleya@gmail.com>
To: river-dev@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi Patricia

Is there perhaps a solid baseline to test against, for example Jini
2.1 to see how many pass/fails we get?

Thanks for all the hard work
Patrick

On Tue, Aug 24, 2010 at 9:58 PM, Patricia Shanahan <pats@acm.org> wrote:
> I ran a batch of the previously ignored QA tests overnight. I got 156 passes
> and 64 failures. This is nowhere near as bad as it sounds, because many of
> the failures were clusters of related tests failing in similar ways,
> suggesting a single problem affecting the base infrastructure for the test
> category. Some of the failures may relate to the known regression that Peter
> is going to look at this week.
>
> Also, it is important to remember that the bugs may be in the tests, not in
> the code under test. A test may be obsolete, depending on behavior that is
> no longer supported.
>
> I do think there is a good enough chance that at least one of the failures
> represents a real problem, and an opportunity to improve River, that I plan
> to start a background activity looking at failed tests to see what is going
> on. The objective is to do one of three things for each cluster of failures:
>
> 1. Fix River.
>
> 2. Fix the test.
>
> 3. Decide the test is unfixable, and delete it. There is no point spending
> disk space, file transfer time, and test load time on tests we are never
> going to run.
>
> Running the subset I did last night took about 15 hours, but that included a
> lot of timeouts.
>
> Patricia
>