lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Updated: (LUCENE-1709) Parallelize Tests
Date Sun, 21 Feb 2010 13:47:28 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael McCandless updated LUCENE-1709:
---------------------------------------

    Attachment: runLuceneTests.py

I'm attaching the scary python script that I use to use multiple
threads when running the tests.

It's not really generic at all.  It's got hardwired paths to my home
dir, it symlinks build/test to /tmp (= tmpfs on my solaris box), it
applies a scary patch to the build xml files (and that patch is
depending on the branch -- only works on 2.9, 3.0, trunk, flex now).

Often the patch fails to apply (as we change the build xml files) so I
have to go and redo them.

Sometimes tests have false positive failure because they temporarily
fill up the tmpfs; if you ctrl+C the test it may leave turd processes
running; etc.

So I wouldn't recommend using this as is.... but maybe someone who is
good-with-the-ant can take the general idea here and make it work more
generally with only ant.

Roughly all that I do is have each thread run its own -Dtestpackage.
The existing search & index packages are too big, so I split them into
2.  I also roughly ordered the tests so that they are "balanced", so
that as each thread pulls from the work queue, the "roughly" fininish
at the same time.  The test-contrib is run by a single thread.

With all the speedups, this is what the output looks like:
{noformat}
[TRUNK]
0 [0:00:00.310867]: run "ant compile-backwards compile-core compile-demo jar-core compile-test
build-contrib"...
0 [0:00:30.612062]: run "ant test-contrib"...
1 [0:00:30.617522]: run "ant test-core -Dtestpackagea=index"...
2 [0:00:30.623840]: run "ant test-core -Dtestpackageb=index"...
3 [0:00:30.630072]: run "ant test-backwards -Dtestpackagea=index"...
4 [0:00:30.637612]: run "ant test-backwards -Dtestpackageb=index"...
5 [0:00:30.645334]: run "ant test-core -Dtestpackagea=search"...
4 [0:00:58.937803]: run "ant test-core -Dtestpackageb=search"...
2 [0:00:59.052593]: run "ant test-backwards -Dtestpackagea=search"...
5 [0:01:31.313156]: run "ant test-backwards -Dtestpackageb=search"...
2 [0:01:52.617814]: run "ant test-core -Dtestpackage=store"...
4 [0:02:01.698477]: run "ant test-backwards -Dtestpackage=store"...
2 [0:02:10.818047]: run "ant test-core -Dtestpackage=analysis"...
3 [0:02:14.808217]: run "ant test-backwards -Dtestpackage=analysis"...
1 [0:02:15.786972]: run "ant test-core -Dtestpackageroot=lucene"...
4 [0:02:21.937936]: run "ant test-backwards -Dtestpackageroot=lucene"...
1 [0:02:25.898000]: run "ant test-core -Dtestpackage=util"...
2 [0:02:31.037923]: run "ant test-backwards -Dtestpackage=util"...
4 [0:02:31.038090]: run "ant test-core -Dtestpackage=document"...
3 [0:02:32.007975]: run "ant test-backwards -Dtestpackage=document"...
5 [0:02:38.017968]: run "ant test-core -Dtestpackage=queryParser"...
3 [0:02:40.097929]: run "ant test-backwards -Dtestpackage=queryParser"...
4 [0:02:40.151166]: DONE
0 [0:02:41.927786]: DONE
1 [0:02:43.077980]: DONE
2 [0:02:46.198287]: DONE
5 [0:02:49.168172]: DONE
3 [0:02:50.197936]: DONE

DONE: took 0:02:51.046643 [528 tests]
{noformat}


> Parallelize Tests
> -----------------
>
>                 Key: LUCENE-1709
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1709
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4.1
>            Reporter: Jason Rutherglen
>             Fix For: 3.1
>
>         Attachments: runLuceneTests.py
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.  
> This task from ANT can be used: http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion: http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message