gump-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Simons <m...@leosimons.com>
Subject Re: Parallelism (was Re: A Few Plans)
Date Mon, 21 Jun 2010 11:19:12 GMT
On 6/21/10 5:34 AM, Stefan Bodewig wrote:
>> This would allow for a little more concurrency than we can do on the
>> Zone or VM... of course we'd have to be able to address all of those
>> cores.  Wonder whether Python has glue for Grand Central Dispatch?

Most of the weight in gump runs is inside the java processes; the other 
half of the latency is svn checkouts/updates.

For the former, you'd need (a) the JVM to hook up GCD (which is for 
apple to do) and (b) maven to do more stuff in parallel (which is on the 
charts for maven 3 I think).

For the latter, IIRC we have code to run more checkouts in parallel, but 
the code is buggy in gump2; and would mean a load increase on the svn 
server which may not be a good thing?

> If you log into one of the machines while Gump is running, the system
> feels sluggish and any opration that hits the file system takes ages
> which makes me fear we are I/O bound rather than CPU bound - making
> those cores do more may not help too much in that case.  I can certainly
> be wrong.

You're absolutely right. Builds are almost always I/O bound and you'll 
see a lot of CPU is actually iowait -- so the numbers in top are usually 
misleading and most of CPU busy-ness is due to overhead of waiting for IO.

> IIRC Gump's trunk supports parallel SCM checkouts but we've restricted
> it to a maximum of one updater because Adam saw problems - it's been a
> long time.

Oh, eh, yep, so that's what I remember too :)

In any case to take advantage of multicores it'd be good to re-implement 
parallelism using python's multiprocess module.

> Currently we don't support building things in parallel at all.  Starting
> several Ant or make builds in parallel would likely do what you expect,
> but I don't know how mvn would deal with multiple processes accessing
> the same local repository (and writing to it) in parallel.

I don't think there's any special code for it in maven, but nevertheless 
I've never really seen an issue with that. A common hudson deployment is 
to run 4-5 builds in parallel using a 'hudson' user that has one local 
repository.


cheers,


Leo

PS: no I'm not dead just persistently e-mail overloaded :)

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org


Mime
View raw message