lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <ser...@gmail.com>
Subject Parallel tests in Benchmark
Date Thu, 01 Apr 2010 04:03:49 GMT
Hi

I'd like to summarize a discussion I had w/ Robert and Mike last night on
IRC, about the parallelism of tasks in Benchmark:

For some reason, ever since parallel tasks were introduced, when I run 'ant
test' from the contrib/benchmark folder (or the root), the tests just hang
at some point, after WriteLineDocTaskTest finishes. What's very weird is
that it seems I'm the only one experiencing this, and so for a long time I
thought it's just a problem w/ my environment ... until yesterday when I did
a fresh checkout of trunk, to a fresh folder and project, and still the
tests stuck.

Thread dump does not show anything relevant to Lucene code, but rather to
Ant. The main thread is waiting on
org/apache/tools/ant/taskdefs/Parallel.spinThreads, another on
org/apache/tools/ant/taskdefs/Execute.waitFor and two other on
java/io/FileInputStream.read. But nothing is related to Lucene code,
directly. Also annoyingly, but conveniently for debugging that issue, it
happens very consistently on my machine - sometimes the test passes, but 90%
hangs.
Running w/ -Drunsequential=1 consistently succeeds.

We've explored different ways to understand the cause of the problem, and
came across several improvements and a workaround, but unfortunately not to
a definite resolution:

* As a last resort, we can add runsequential property to benchmark
build.xml, which forces Benchmark tests to run sequentially. Since that's a
tiny package which takes a few seconds to run anyway, and parallelism
doesn't improve much (it actually runs slower, when it passes, on my
machine: parallel=15 sec, seq=11 sec), this might be acceptable.

* Moving the junit temp files (such as that flag file) created to the temp
directory each test uses. This is actually a good thing to do anyway (thanks
Robert for spotting that), because it avoids accidental commits of such
files :), as well as doesn't clutter the main environment. We've done that
because when I hit CTR:+C to stop one of the runs which hung, we received a
FNFE on a junit flag "file is being accessed by another process" (something
like that), and thought this is related to the hangs I'm seeing. Anyway,
this file is attempted access by multiple JVMs concurrently, which seems
bad.

* Explore the JUnit Formatter code under src/test, since it uses file
locking. I've disabled locks (using NoLockFactory), however the test still
hung.

* Change common-build.xml threadsPerProcessor to '1' instead of '2'. We
think that might be a good thing to do anyway - if people run on machines
with just one CPU, threading is not expected to help much, as opposed to
running on multiple CPUs. But we don't want to enforce it on anyone, so we
think to change the default to '1', but introduce a property
'threadsPerProcessor' which users will be able to set explicitly.
** Surprisingly, when I set it to '1' or '10' (I run on dual-core Thinkpad
W500), the test consistently passes - it just doesn't like the value '2'. At
least it passed as long as I ran it, maybe a thread hang is lurking for me
around the corner somewhere.

* We made sure the benchmark tests indeed read/write the test data files
from/to unique directories. But like I said - there is no hang in Lucene
code reported in the thread dump.

It was very late last night when we stopped, and my eyes were tired, so I
didn't summarize it right away. Robert, I hope I've captured everything we
did, if not please add.

Anyone's got any suggestions? It's unfortunate that I'm the only one running
into this problem, because whatever the suggestions are, you'll probably
need me to confirm them :). And I'm going away for 3 days (camping - no
internet ... well at least no laptop :)), so unless someone has a suggestion
within the coming few hours, we can continue that when I get back.

Shai

Mime
View raw message