lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@cs.put.poznan.pl>
Subject Re: [JENKINS] Lucene-Solr-tests-only-trunk - Build # 13992 - Failure
Date Sun, 13 May 2012 05:43:31 GMT
> The problem is ... that test line produces an enormous amount of
> output... the events file is 11 GB.

Yes, this is the problem, not the OOM in the test itself.

> ... it looks like the parent java process is OOMEing trying to gather
> up all the output.

Yes it currently attempts to buffer the output as part of the test
completion event to report listeners. So indeed, this might have OOMed
on the parent.

> Is there some way to not buffer the output with "ant test"...?

Not really, this is by-design. The issue here is that we have multiple
forked JVMs and all of them may be writing to sysouts at once.
Listeners (including text output listener that writes to ANT console)
receive aggregated events that are atomic view of tests' execution
(for example a single test, with its status, output, timing, etc.).
API wise this could be in fact done at the runner level when
individual events are received (and assuming a single JVM is forked)
but it just smells bad to me.

It is also possible to run just the forked JVM and get its raw output
(much like Mike does with the python runner now?), but it's again
hacky.

Large sysout buffers have occurred in the past (Yonik asked for it for
example). It's definitely possible but it adds complexity to the
runner (it'd have to spill to disk and events would then just point at
a section of the output file). I'll do it at some point but I don't
consider it super-urgent. To my defense, I think the default JUnit
runner would also OOM with large outputs because the mechanism of
passing output data to listeners is memory-based as well:

http://www.jajakarta.org/ant/ant-1.6.1/docs/ja/manual/api/org/apache/tools/ant/taskdefs/optional/junit/JUnitResultFormatter.html#setSystemOutput(java.lang.String)

I filed LUCENE-4053 to track this. I'll grind the alternatives for a
while and think of a way to integrate it.

Dawid

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message