db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Embretsen <John.Embret...@Sun.COM>
Subject Re: Cleaning up test failures
Date Fri, 20 Jan 2006 13:14:43 GMT
David W. Van Couvering wrote:

> I would like to have an email list set up that sends out test 
> results, for those of us who would like a "push" model to see what's 
> going on. Personally I would want to see test failures sent to 
> derby-dev.  If you don't like seeing them, you can filter them out, 
> but I think it's good for the developers to have awareness of what's 
> going on.
> 
> I think that combined with logging JIRA issues would be good.  We 
> could use Kathey's idea to get a report on open test bugs by 
> filtering on the right component.

If it is agreed upon that we need to change the way developers get
alerted about test failures, I basically support what what is quoted
above. I've seen many good suggestions the past 24 hours for how to
increase awareness and potentially trigger action when a test in
derbyall fails. However....

With regards to the _awareness_ part of the challenge, what we need
(IMHO) more than yet another wiki-page or other kind of web page that
people have to manually update and/or navigate to, is some kind of
"push" mechanism such as automatically sending e-mails to some mailing list.

Adding JIRA-entries sounds good to me, since e-mails are sent to
derby-dev when issues are created/updated (although this strategy seemed
a bit chaotic to me yesterday), thus increasing awareness. But, it
requires a certain amount of *manual* work. Someone has to have the itch
to check the actual test results (whether they are provided by Sun or
IBM or someone else), pick a test failure (if any exist), check if it
has been reported in JIRA already, and, if not, create a new JIRA issue,
etc. Because of this, I do not think this strategy is sufficient as a
long-term solution (but it may be a part of it).

I suggest that we focus on deciding:

  a) Do we want to get automated e-mails reporting test results?

  b) If so, how often should it be reported (after every test run, after
   every test run with failures, after every test run with a certain
number of failures, once a day, once a day if failures occured, ...)?
(I guess the number of sources producing public test reports regularly
may be a factor here in the long run, but today there is only one such
source)

  c) Where should such e-mails be sent (derby-dev, derby-test-results,
derby-commits, ...)?

...and that we try to keep this separate from the discussions of adding
new web-pages with test results, wiki-pages with test results, new
JIRA-types, etc.


-- 
John








Mime
View raw message