Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <1786429646.1259516420617.JavaMail.jira@brutus>
Date: Sun, 29 Nov 2009 17:40:20 +0000 (UTC)
From: "Erick Erickson (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-2096) Investigate parallelizing Ant junit
 tests
In-Reply-To: <497419657.1259354360727.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783436#action_12783436 ] 

Erick Erickson commented on LUCENE-2096:
----------------------------------------

Parallelizing tests is proving trickier than I'd hoped. Part of the problem is my not-wonderful ant skills...

But what I've found so far with trying to use ForEach is that stuff gets in the way. In particular, the <sequential> tag in the test-macro body I'm pretty sure defeats any parallelizing attempts by ForEach. Taking it out isn't straightforward.

In some of my experiments, I got tests to fire off in parallel, but then started running into wonky errors that were so strange now I can't remember them, but some having to do with what looked like file contention for some temporary test files.

Googling around I think I remember posts by Jason Ruthgren trying to so something similar in SOLR (?). Jason: if I'm remembering right did you find any joy?

Then we'd have to rework how success and failure are handled because there's contention for that file as well.

Now I'm wondering if the "scary python script" gets us more bang for the buck. I wrote a Groovy script the probably is a near-cousin for experiments and I'm wondering what would happen if we wrote a special testcase-type target that did NOT depend upon compile-test or, really, much of anything else and counted on the user to make sure to build the system first before using whatever script wecame up with. We don't really lose functionality by recursively looking for Test*.java files because that's what's done internally in the build files anyway. So doing that outside or inside the ant files doesn't seem like a loss.

I'm putting this in the JIRA issue to preserve it for posterity. Meanwhile, I'll appeal to Ant gurus if they want to try whacking the Ant build files, and see what the script notion brings...


> Investigate parallelizing Ant junit tests
> -----------------------------------------
>
>                 Key: LUCENE-2096
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2096
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Build
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>            Priority: Minor
>
> Ant Contrib has a "ForEach" construct that may speed up running all of the Junit tests by parallelizing them with a configurable number of threads. I envision this in several stages. First, see if ForEach works for us with hard-coded lists, distribute this for testing then make the changes "for real". I intend to hard-code the list for the first pass, ordered by the time they take. This won't do for check-in, but will give us a fast proof-of-concept.
> This approach will be most useful for multi-core machines.
> In particular, we need to see whether the parallel tasks are isolated enough from each other to prevent mutual interference.
> All this assumes the fragmentary reference I found is still available...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org