maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Kulp <dk...@apache.org>
Subject Re: Parallel vs weave
Date Fri, 11 Jun 2010 19:02:36 GMT
On Friday 11 June 2010 1:40:07 pm Kristian Rosenvold wrote:
> Interesting info, and good numbers! Weave is a double edged sword;
> while it may be promising even
> faster builds, by design it has this irritating tendency to run mojos
> (of the same execution in different modules)
> *exactly* in parallel, more or less simulating how I'd write a unit
> test to test thread safety. So weave
> has proved to find almost *all* of the concurrency bugs I've located
> (where parallel doesn't have any problems).
> There are also a couple of minor details regarding the actual
> execution planning in weave mode that are still
> somewhat open - it's a much more complex execution model than
> parallel; but I'm concentrating on
> parallel and the ecosystem in general ATM and just making sure weave
> works for every project I test.
> 
> Antrun itself is threadsafe, but whatever is being executed there is
> another matter. If this is committed to cxf
> trunk I can/will look at it once I get through the known issues.

Yep.   To reproduce

1) checkout CXF trunk from:
http://svn.apache.org/repos/asf/cxf/trunk/

2) Build everything (non-parallel right now)
mvn -Pnochecks -Dmaven.test.skip.exec=true install
That gets everything set and all the code generation done and such.  (I'm 
pretty sure not all the code gen plugins are threadsafe yet, but they all "do 
nothing" if they don't have to do anything)

Then just run:
mvn -Pnochecks -T 12W 
or similar.     For me, 100% of the time it barfs in antrun in the testutils 
module.

You need the -Pnochecks right now when using -T until we get a release of 
checkstyle that is threadsafe.   Alternatively, update the parent/pom.xml to 
use the snapshots of checkstyle and pmd.

Dan



> 
> Kristian
> 
> On Fri, Jun 11, 2010 at 6:57 PM, Daniel Kulp <dkulp@apache.org> wrote:
> > Well, weave mode doesn't work for CXF.   Ends up with a
> > NullPointerException down in antrun-plugin someplace.   Thus, I cannot
> > help you there.
> > 
> > For CXF on my i7 820QM (4 cores, 8 threads) turning off checkstyle/pmd
> > and having everything already code generated/compiled (so mostly just
> > running tests)
> > 
> > Linear: 26 minutes
> > Parallel: 9 minutes  (-T 12 is what I'm using)
> > 
> > Dan
> > 
> > On Friday 11 June 2010 12:24:13 pm Kristian Rosenvold wrote:
> >> I am picking up some positive tweets here and there, but I'd be really
> >> happy if those of you who test parallel building could report back with
> >> some numbers in response to this message. I am particularly interested
> >> in the difference between parallel and weave (and linear too), mostly to
> >> assess that weave is worth the effort.
> >> 
> >> So after tweaking around with threads a bit to find out what runs
> >> fastest I'd really enjoy reports like this one (real data):
> >> 
> >> Linear: 50 seconds
> >> Parallel: 40 seconds
> >> Weave: 29 seconds
> >> For a 10 module project using -T 1C on a 6 core box.
> >> 
> >> 
> >> For those wondering:
> >> * Both parallel and weave favorize projects with lots of modules
> >>   and "wide" dependency graphs.
> >> * Weave is /probably/ quite a lot faster than parallel for projects
> >>   with lots of tests fairly evenly distributed among the modules.
> >> * Putting all your tests in one module is a bad idea in this context.
> >> * Some mojos (like war/ear) can "dwarf" all the others,
> >>   effectively limiting the overall potential somewhat.
> >> 
> >> Kristian
> >> 
> >> 
> >> 
> >> 
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> > 
> > --
> > Daniel Kulp
> > dkulp@apache.org
> > http://dankulp.com/blog
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org

-- 
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message