db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David W. Van Couvering" <David.Vancouver...@Sun.COM>
Subject Re: Cleaning up test failures
Date Fri, 20 Jan 2006 18:19:18 GMT

Mike Matrigali wrote:
> there are 2 issues, email and JIRA tracking.
> I don't have much to say to the email discussion.  I find that if
> some email gets sent every day I am going to ignore it unless I run
> my tests and have a problem.  In that case I can either check the
> email or check a web site, so either solution is fine.  The most
> useful daily email to me would be a 1 line subject that says N nightly
> failures across all platforms, and maybe a pointer in the email to
> find more info.

Are you amenable to a derby-test-results list and emails to derby-dev 
when there is a failure?  The idea would be that these failure emails 
would be RARE, otherwise we have a problem :)

> As to the JIRA issue, I still would prefer a new category just because
> I want it very easy to query.  I think even the below proposal will
> not allow for querying of just nightly test failures as there are bugs
> that could (though I agree probably unlikely), fall into the proposed
> classification and not be nightly test errors.  If jira supported
> some sort of "sub-category" that would be perfect.  Does anyone feel
> strongly enough to vote -1 on a new component?

No, I actually think your idea makes sense, it would help separate out 
bugs in the tests to actual test failures.

> To the guidelines it would be good to encourage people to add comments
> to the bug in the case of intermittent failures, especially if they
> see the problem in a different environment than has already been
> proposed.  Sometimes these can be a pain to track down and any evidence
> you can gather helps, especially if the developer can never make it
> happen in the environments that he has access to.  The comments can
> provide a historical trace of how often it is happening and where.

Maybe we need a Wiki page that contain these guidelines.  I guess since 
I brought this up I can work on that.

> Process wise what should be done about an issue like myrna's DERBY-804.
> A "fix" has been made so that nightly tests won't fail, by not running
> the test.  This is a reasonable first step, and sometimes for old JVM's 
> it is likely we just won't ever fix the test.  One option is that the
> bug gets moved off of the derby nightly testing component and into the
> test category with an appropriate severity level setting (in this case 
> likely low).

I think that your idea makes sense, move the bug off nightly tests 
component to tests, and assign it an appropriate category.

> David W. Van Couvering wrote:
>> Great email, John.  May I summarize a potential proposal.  Do people 
>> feel we need a vote on this?
>> - Create a new alias, derby-test-results, that send *all* test results
>> - Send test *failures* to derby-dev
>> - Have a general guideline that test failures should be logged under 
>> the Test component with a high priority (I am using Critical for 
>> cross-platform issues and Major for single-platform issues).
>> - Ultimately committers are responsible for maintaining the quality of 
>> the codeline and they may take further action where necessary (such as 
>> vetoing any checkins except those related to test fixes).
>> Sound good?
>> David
>> John Embretsen wrote:
>>> David W. Van Couvering wrote:
>>>> I would like to have an email list set up that sends out test 
>>>> results, for those of us who would like a "push" model to see what's 
>>>> going on. Personally I would want to see test failures sent to 
>>>> derby-dev.  If you don't like seeing them, you can filter them out, 
>>>> but I think it's good for the developers to have awareness of what's 
>>>> going on.
>>>> I think that combined with logging JIRA issues would be good.  We 
>>>> could use Kathey's idea to get a report on open test bugs by 
>>>> filtering on the right component.
>>> If it is agreed upon that we need to change the way developers get
>>> alerted about test failures, I basically support what what is quoted
>>> above. I've seen many good suggestions the past 24 hours for how to
>>> increase awareness and potentially trigger action when a test in
>>> derbyall fails. However....
>>> With regards to the _awareness_ part of the challenge, what we need
>>> (IMHO) more than yet another wiki-page or other kind of web page that
>>> people have to manually update and/or navigate to, is some kind of
>>> "push" mechanism such as automatically sending e-mails to some 
>>> mailing list.
>>> Adding JIRA-entries sounds good to me, since e-mails are sent to
>>> derby-dev when issues are created/updated (although this strategy seemed
>>> a bit chaotic to me yesterday), thus increasing awareness. But, it
>>> requires a certain amount of *manual* work. Someone has to have the itch
>>> to check the actual test results (whether they are provided by Sun or
>>> IBM or someone else), pick a test failure (if any exist), check if it
>>> has been reported in JIRA already, and, if not, create a new JIRA issue,
>>> etc. Because of this, I do not think this strategy is sufficient as a
>>> long-term solution (but it may be a part of it).
>>> I suggest that we focus on deciding:
>>>  a) Do we want to get automated e-mails reporting test results?
>>>  b) If so, how often should it be reported (after every test run, after
>>>   every test run with failures, after every test run with a certain
>>> number of failures, once a day, once a day if failures occured, ...)?
>>> (I guess the number of sources producing public test reports regularly
>>> may be a factor here in the long run, but today there is only one such
>>> source)
>>>  c) Where should such e-mails be sent (derby-dev, derby-test-results,
>>> derby-commits, ...)?
>>> ...and that we try to keep this separate from the discussions of adding
>>> new web-pages with test results, wiki-pages with test results, new
>>> JIRA-types, etc.

View raw message