From harmony-dev-return-9703-apmail-incubator-harmony-dev-archive=incubator.apache.org@incubator.apache.org Wed Jul 05 05:41:02 2006 Return-Path: Delivered-To: apmail-incubator-harmony-dev-archive@www.apache.org Received: (qmail 16125 invoked from network); 5 Jul 2006 05:41:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 5 Jul 2006 05:41:01 -0000 Received: (qmail 72850 invoked by uid 500); 5 Jul 2006 05:40:58 -0000 Delivered-To: apmail-incubator-harmony-dev-archive@incubator.apache.org Received: (qmail 72797 invoked by uid 500); 5 Jul 2006 05:40:58 -0000 Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: harmony-dev@incubator.apache.org Delivered-To: mailing list harmony-dev@incubator.apache.org Received: (qmail 72786 invoked by uid 99); 5 Jul 2006 05:40:58 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jul 2006 22:40:58 -0700 X-ASF-Spam-Status: No, hits=0.5 required=10.0 tests=DNS_FROM_RFC_ABUSE,HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of ivavladimir@gmail.com designates 66.249.82.195 as permitted sender) Received: from [66.249.82.195] (HELO wx-out-0102.google.com) (66.249.82.195) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jul 2006 22:40:57 -0700 Received: by wx-out-0102.google.com with SMTP id s6so626654wxc for ; Tue, 04 Jul 2006 22:40:36 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=i3LckMal+/O0erqNGzVhQlWW1jKYlVYRXYaeRAcAkIZvzWYrUYmIDVVzSuqi4RlcHpEFc/oW7Oy1hHRTMaRLsEaxLBvulUFbEs/RuOSRWHg/eUUBuFbzBgAsL8j6WvDgtRH4fyz4P/MJvubgf8kFwN8zcK9BTWYiHnVlw16ZfUY= Received: by 10.70.59.17 with SMTP id h17mr8489378wxa; Tue, 04 Jul 2006 22:40:36 -0700 (PDT) Received: by 10.70.26.5 with HTTP; Tue, 4 Jul 2006 22:40:36 -0700 (PDT) Message-ID: <7273946b0607042240h7a91bef6n45c2a4e03af6c250@mail.gmail.com> Date: Wed, 5 Jul 2006 12:40:36 +0700 From: "Vladimir Ivanov" To: harmony-dev@incubator.apache.org Subject: Re: [classlib][testing] excluding the failed tests In-Reply-To: <44A984EF.1070202@gmail.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_35166_11966588.1152078036269" References: <000c01c69ec8$22b4cd40$f542fb0a@LITTLEGUY> <44A984EF.1070202@gmail.com> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N ------=_Part_35166_11966588.1152078036269 Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Yesterday I tried to add a regression test to existing in security module TestCase, but, found that the TestCase is in exclude list. I had to un-exclude it, run, check my test passes and exclude the TestCase again =96= it was a little bit inconvenient, besides, my new valid (I believe) regression test will go directly to exclude list after integration... I see that we are near to decision what to do with failing tests. Am I right that we are at the point of agreement on the following?: There could be two groups of failing tests: *Tests that never passed. *Tests that recently started failing. Test that never passed should be stored in TestCases with suffix "Fail" ( StringFailTest.java for example). They are subject for review and either deletion or fixing or fixing implementation if they find a bug in API implementation. There should be 0 tests that recently started failing. If such test appears it should be fixed within 24h, otherwise, commit which introduced the failure will be rolled back. Right? Thanks, Vladimir On 7/4/06, Tim Ellison wrote: > Nathan Beyer wrote: > > Based on what I've seen of the excluded tests, category 1 is the > predominate > > case. This could be validated by looking at old revisions in SVN. > > I'm sure that is true, I'm just saying that the build system 'normal' > state is that all enabled tests pass. My concern was over your > statement you have had failing tests for months. > > What is failing for you now? > > Regards, > Tim > > > >> -----Original Message----- > >> From: Geir Magnusson Jr [mailto: geir@pobox.com] > >> > >> Is this the case where we have two 'categories'? > >> > >> 1) tests that never worked > >> > >> 2) tests that recently broke > >> > >> I think that a #2 should never persist for more than one build > >> iteration, as either things get fixed or backed out. I suppose then w= e > > >> are really talking about category #1, and that we don't have the > "broken > >> window" problem as we never had the window there in the first place? > >> > >> I think it's important to understand this (if it's actually true). > >> > >> geir > >> > >> > >> Tim Ellison wrote: > >>> Nathan Beyer wrote: > >>>> How are other projects handling this? My opinion is that tests, whic= h > > >> are > >>>> expected and know to pass should always be running and if they fail > and > >> the > >>>> failure can be independently recreated, then it's something to be > >> posted on > >>>> the list, if trivial (typo in build file?), or logged as a JIRA > issue. > >>> Agreed, the tests we have enabled are run on each build (hourly if > >>> things are being committed), and failures are sent to commit list. > >>> > >>>> If it's broken for a significant amount of time (weeks, months), the= n > > >> rather > >>>> than excluding the test, I would propose moving it to a "broken" or > >>>> "possibly invalid" source folder that's out of the test path. If it > >> doesn't > >>>> already have JIRA issue, then one should be created. > >>> Yes, though I'd be inclined to move it sooner -- tests should not sta= y > > >>> broken for more than a couple of days. > >>> > >>> Recently our breakages have been invalid tests rather than broken > >>> implementation, but they still need to be investigated/resolved. > >>> > >>>> I've been living with consistently failing tests for a long time now= . > > >>>> Recently it was the unstable Socket tests, but I've been seeing the > >> WinXP > >>>> long file name [1] test failing for months. > >>> IMHO you should be shouting about it! The alternative is that we > >>> tolerate a few broken windows and overall quality slips. > >>> > >>>> I think we may be unnecessarily complicating some of this by assumin= g > > >> that > >>>> all of the donated tests that are currently excluded and failing are > >>>> completely valid. I believe that the currently excluded tests are > >> either > >>>> failing because they aren't isolated according to the suggested test > >> layout > >>>> or they are invalid test; I suspect that HARMONY-619 [1] is a case o= f > > >> the > >>>> later. > >>>> > >>>> So I go back to my original suggestion, implement the testing > proposal, > >> then > >>>> fix/move any excluded tests to where they work properly or determine > >> that > >>>> they are invalid and delete them. > >>> Yes, the tests do need improvements too. > >>> > >>> Regards, > >>> Tim > >>> > >>> > >>>> [1] https://issues.apache.org/jira/browse/HARMONY-619 > >>>> > > > > > > > > --------------------------------------------------------------------- > > Terms of use : http://incubator.apache.org/harmony/mailing.html > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org > > > > > > -- > > Tim Ellison ( t.p.ellison@gmail.com) > IBM Java technology centre, UK. > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org > For additional commands, e-mail: harmony-dev-help@incubator.apache.org > > ------=_Part_35166_11966588.1152078036269--