db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David W. Van Couvering" <David.Vancouver...@Sun.COM>
Subject Re: Cleaning up test failures
Date Fri, 20 Jan 2006 17:04:42 GMT
Great email, John.  May I summarize a potential proposal.  Do people 
feel we need a vote on this?

- Create a new alias, derby-test-results, that send *all* test results

- Send test *failures* to derby-dev

- Have a general guideline that test failures should be logged under the 
Test component with a high priority (I am using Critical for 
cross-platform issues and Major for single-platform issues).

- Ultimately committers are responsible for maintaining the quality of 
the codeline and they may take further action where necessary (such as 
vetoing any checkins except those related to test fixes).

Sound good?


John Embretsen wrote:
> David W. Van Couvering wrote:
>> I would like to have an email list set up that sends out test results, 
>> for those of us who would like a "push" model to see what's going on. 
>> Personally I would want to see test failures sent to derby-dev.  If 
>> you don't like seeing them, you can filter them out, but I think it's 
>> good for the developers to have awareness of what's going on.
>> I think that combined with logging JIRA issues would be good.  We 
>> could use Kathey's idea to get a report on open test bugs by filtering 
>> on the right component.
> If it is agreed upon that we need to change the way developers get
> alerted about test failures, I basically support what what is quoted
> above. I've seen many good suggestions the past 24 hours for how to
> increase awareness and potentially trigger action when a test in
> derbyall fails. However....
> With regards to the _awareness_ part of the challenge, what we need
> (IMHO) more than yet another wiki-page or other kind of web page that
> people have to manually update and/or navigate to, is some kind of
> "push" mechanism such as automatically sending e-mails to some mailing 
> list.
> Adding JIRA-entries sounds good to me, since e-mails are sent to
> derby-dev when issues are created/updated (although this strategy seemed
> a bit chaotic to me yesterday), thus increasing awareness. But, it
> requires a certain amount of *manual* work. Someone has to have the itch
> to check the actual test results (whether they are provided by Sun or
> IBM or someone else), pick a test failure (if any exist), check if it
> has been reported in JIRA already, and, if not, create a new JIRA issue,
> etc. Because of this, I do not think this strategy is sufficient as a
> long-term solution (but it may be a part of it).
> I suggest that we focus on deciding:
>  a) Do we want to get automated e-mails reporting test results?
>  b) If so, how often should it be reported (after every test run, after
>   every test run with failures, after every test run with a certain
> number of failures, once a day, once a day if failures occured, ...)?
> (I guess the number of sources producing public test reports regularly
> may be a factor here in the long run, but today there is only one such
> source)
>  c) Where should such e-mails be sent (derby-dev, derby-test-results,
> derby-commits, ...)?
> ...and that we try to keep this separate from the discussions of adding
> new web-pages with test results, wiki-pages with test results, new
> JIRA-types, etc.

View raw message