xml-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shane_Curc...@lotus.com
Subject Re: Test Infrastructure Project Proposal - re:loggingLevels
Date Tue, 13 Feb 2001 11:58:51 GMT
(Only broadcasting on general@xml.apache.org)

Ah-ha! (got a mini epiphany from your note)

---- you "Sean Kelly" <kelly@ad1440.net> wrote: ----
> > Incomplete/Pass/Fail/Error/Ambiguous.
> In my experience, this many kinds of logging
> is counterproductive.  Unit tests should either
> succeed or fail---nothing more.
Ah - you said unit test.  The stuff I use in Xalan is meant to be a general
automated testing framework, not just for unit tests.  I think with the
proper doc and samples it will work well for developer-centric unit
testing, but I know from experience that it works well with larger groups
of quality engineers writing test scripts.

Now there's a separate discussion as to how many open source types here
would potentially classify themselves as 'quality engineers' at one time or
another, but for the time being, that's what I consider myself.

> Developers will be more likely to run the tests
> more often if they don't have to spend time
> interpreting test logs.  ("Ambiguous"?  What,
> the system thought that *maybe* the test passed?)
Yes, the logs need to be clear - and within those values, they are.  If
testwriters don't want to use the other values, they don't need to, but I
find it very useful in Xalan.

One minimum requirement (IMO) for a test harness is a feature that
automatically calculates the final result and presents it in a rolled up
fashion.  In the many times the tests passes, you're done.  When something
doesn't pass, you should be able to get a detailed report of just the items
that didn't pass with basic descriptions of what each test point did, so
you can start looking at the code.  Note that in my mind, these may well be
larger scale functional or integration or system tests, not just individual
API unit tests.

Oh, and for the curious, the meanings of the Xalan test's return values are
discussed in http://xml.apache.org/xalan-j/test/design.html  a few
paragraphs down (not my best writing, but it does the trick). AMBG is
neither pass nor fail, because the expected or gold result was not
available at testrun time to compare to.  Quite useful when adding new
tests that you haven't been able to really verify the outputs of yet.

- Shane

View raw message