db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francois Orsini <francois.ors...@gmail.com>
Subject Re: Failures in latest revisions
Date Thu, 19 Jan 2006 01:02:56 GMT
I'd say that either there is a qualified and expected *clean* run of
DerbyAll or none - I know this sounds like stating the obvious either I
would expect my derbyall to run clean _or_ I know the errors are expected
(documented somewhere and in the canon's even better or jUnit equivalent) -
if none are met, then I'd go and spend time as far as why am not getting the
expected results - then I would very likely want to compare with another
source of test results (i.e. What Ole has produced) and see if the errors am
seeing are due to my changes or not - does not mean, one should not look
into them, but at least you'd know that you have not regressed DerbyAll with
your changes. Personally if I see a diff in DerbyAll and it is not obvious
that it is not expected, I then would want to compare with another source,
run a DerbyAll without my changes or ask the community - if I can rely on
some external (and reliable) source, then I don't have to run DerbyAll again
*without* my changes (as you know it takes quite some time to run it until
completion). - I know we can't expect Ole's page to be there forever but at
the same time, it'd be nice to document it somewhere for the moment in some
README or guidelines content. But of course, someone has to scratch that
itch ;)

My 0.02 cents,

--francois

On 1/18/06, David W. Van Couvering <David.Vancouvering@sun.com> wrote:
>
> I am not suggesting regular emails.  But I also doubt that developers,
> who are all quite busy, are going to remember to regularly check the
> results.
>
> What I'd like to see is somebody who volunteers to monitor the page and
> send an email out if things are getting out of hand.  I would leave it
> to the judgement of the volunteer to identify what "out of hand" means.
>   Personally, if more than three to five tests are failing in derbyall
> then I think we have a problem.  Or if we are seeing a general increase
> in failures rather than a decrease.
>
> David
>
> Ole Solberg wrote:
> > David W. Van Couvering wrote:
> >
> >> I got some test failures in derbynetclient mats, so I checked the
> >> tinderbox.  We have quite a number of failures on the latest revision
> >> that ran tests, 370061, running on Solaris 10 x86:
> >>
> >>
> http://www.multinet.no/~solberg/public/Apache/TinderBox_Derby/Limited/testSummary-370061.html
> >>
> >>
> >> derbylang: 1 failure
> >> derbytools: 1 failure
> >> encryptionBlowfish: 1 failure
> >> i18nTest: 1 failure
> >> jdbcapi: 1 failure
> >> derbynetclientmats: 3 failures
> >> derbynetmats: 2 failures
> >> encryptionAll: 2 failures
> >> encryptoin: 1 failure
> >>
> >> derbyall: 12 failures
> >>
> >> This seems a bit much; it's hard for a developer to know if their own
> >> changes are valid as you have to sift through all the existing
> >> failures.  The last Solaris 10 x86 regression test run on revision
> >> 369861 had only 2 failures.
> >>
> >> Looking at the derbyall history, at least on XP it seems to be getting
> >> worse and worse (from 7 failures on 1/6 to 12 failures on 1/17).  On
> >> Solaris 10 it's gone from 1 failure to 5 failures, while on Linux it's
> >> stayed steady.
> >>
> >> Can those of us who have checked in/contributed patches lately please
> >> look at the failures and see if you recognize what might be causing
> them?
> >>
> >> Also: is anybody keeping an eye on these and raising a flag if the
> >> tests start failing?  I thought we had a pretty strict rule that we
> >> should have 100% pass on derbyall.
> >
> >
> > Are you suggesting that we should send an e-mail on derby-dev on the
> > status of the tests? Wouldn't that just drown in the existing volume?
> >
> > Since these test results are available for anyone to see at
> > http://www.multinet.no/~solberg/public/Apache/
> > I was thinking of the 3-three sub-pages as the *flags* !
> >
> > I was hoping that patch submitters and committers would use these pages
> > to check results on other platforms than what they do test on
> themselves.
> >
> >
> > And, of course, if the results on
> > http://www.multinet.no/~solberg/public/Apache/ do not match what other
> > people are getting on the same platforms please tell me, because then I
> > probably have a problem with my test environment!
> >
> >>
> >> Thanks,
> >>
> >> David
> >
> >
> >
>
>
>

Mime
View raw message