tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Burch <br...@pingtoo.com>
Subject Re: Detecting failures of unit tests
Date Thu, 19 Jan 2012 00:01:02 GMT
On 19/01/12 07:12, Christopher Schultz wrote:
> All,
>
> I was testing 7.0.25 and "ant test" reports BUILD SUCCESSFUL but I
> started looking at the TEST-*.txt files that are emitted and I was
> wondering about a few things.
>
> First, I should probably be look at the bottom of the file for the junit
> summary that looks like this:
>
> Testsuite: org.apache.tomcat.util.threads.TestLimitLatch
> Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 2.545 sec
>
> Observing the "Failures: 0, Errors: 0" indicates that the tests have all
> passed correctly, right?
>
> I'm asking because I can also see things like this:
>
> TEST-org.apache.catalina.connector.TestMaxConnections.APR.txt:INFO:
> There were [4] passed requests and [2] connection failures
>
> Obviously, that's an INFO line, but it does indicate a "failure" of some
> kind.
>
> There are also some log lines like this:
>
> TEST-org.apache.catalina.startup.TestListener.NIO.txt:SEVERE: Context
> [/] startup failed due to previous errors
>
> Does that merely indicate that the test itself caused a failure, and
> that the failure-to-startup was intentional?
>
> Similarly:
>
> TEST-org.apache.catalina.tribes.group.interceptors.TestNonBlockingCoordinator.APR.txt:org.apache.catalina.tribes.ChannelException:
> Send failed, and sender is disconnected. Not retrying.; Faulty
> members:tcp://{127, 0, 0, 1}:4005;
>
> Again, all test files say "Failures: 0, Errors: 0" so I guess everything
> is okay. It's just tough to see those log lines without asking.
>
> Thanks,
> -chris

1. I noticed this kind of message when I was looking for problems in my 
own new tomcat tests. Because they didn't originate from my own tests, I 
didn't follow it up. Sorry...

2. I've seen similar situations in other projects, but I don't know if 
my explanation is relevant to the tomcat tests... in all previous cases, 
my tests have been complex enough to require starting another thread so 
I could have both a client and server active during each individual test 
method of the class.

3. The tomcat tests (at least the ones I have worked on) subclass 
TomcatBaseTest, which starts Tomcat in a new thread, which puts them 
into the category I am talking about.

4. As far as I remember, it is possible for these sub-threads to "fail", 
or even interfere with each-other (clashes for port numbers or other 
shared resources), or even interfere with other completely different 
test classes doing the same kind of thing, while the main thread happily 
passes all its junit assertions and reports "success".

5. Forgive me if this is too vague, or too simplistic to be helpful. 
Your comment reminded me of myself (about 10 years ago) when everyone 
else was happily reading successful junit summaries and I started 
looking at the log files generated by the tests. I found the messages 
(like yours) difficult to understand, and even more difficult to 
diagnose - set a debugger break point in the sub-thread and you will 
screw up the timing of the tests, thus never even arriving at the 
troublesome code. I often had to catch the exception when it was thrown, 
then analyse the call stack - just like the old days of reading core dumps!

6. If you are still reading this, then I'll cheer you up by saying about 
5 in 6 of these "problems" came down to artefacts of the test design, 
rather than bugs in the logic actually under test. Unfortunately, I had 
to debug and fix all of them before I could stop worrying!

I hope this war story has been helpful.

Brian



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


Mime
View raw message