lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson (JIRA)" <>
Subject [jira] Commented: (LUCENE-2096) Investigate parallelizing Ant junit tests
Date Sun, 29 Nov 2009 17:40:20 GMT


Erick Erickson commented on LUCENE-2096:

Parallelizing tests is proving trickier than I'd hoped. Part of the problem is my not-wonderful
ant skills...

But what I've found so far with trying to use ForEach is that stuff gets in the way. In particular,
the <sequential> tag in the test-macro body I'm pretty sure defeats any parallelizing
attempts by ForEach. Taking it out isn't straightforward.

In some of my experiments, I got tests to fire off in parallel, but then started running into
wonky errors that were so strange now I can't remember them, but some having to do with what
looked like file contention for some temporary test files.

Googling around I think I remember posts by Jason Ruthgren trying to so something similar
in SOLR (?). Jason: if I'm remembering right did you find any joy?

Then we'd have to rework how success and failure are handled because there's contention for
that file as well.

Now I'm wondering if the "scary python script" gets us more bang for the buck. I wrote a Groovy
script the probably is a near-cousin for experiments and I'm wondering what would happen if
we wrote a special testcase-type target that did NOT depend upon compile-test or, really,
much of anything else and counted on the user to make sure to build the system first before
using whatever script wecame up with. We don't really lose functionality by recursively looking
for Test*.java files because that's what's done internally in the build files anyway. So doing
that outside or inside the ant files doesn't seem like a loss.

I'm putting this in the JIRA issue to preserve it for posterity. Meanwhile, I'll appeal to
Ant gurus if they want to try whacking the Ant build files, and see what the script notion

> Investigate parallelizing Ant junit tests
> -----------------------------------------
>                 Key: LUCENE-2096
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Build
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>            Priority: Minor
> Ant Contrib has a "ForEach" construct that may speed up running all of the Junit tests
by parallelizing them with a configurable number of threads. I envision this in several stages.
First, see if ForEach works for us with hard-coded lists, distribute this for testing then
make the changes "for real". I intend to hard-code the list for the first pass, ordered by
the time they take. This won't do for check-in, but will give us a fast proof-of-concept.
> This approach will be most useful for multi-core machines.
> In particular, we need to see whether the parallel tasks are isolated enough from each
other to prevent mutual interference.
> All this assumes the fragmentary reference I found is still available...

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message