Mailing-List: contact general-help@gump.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Gump code and data" <general@gump.apache.org>
Received-SPF: neutral (athena.apache.org: local policy)
From: Stefan Bodewig <bodewig@apache.org>
To: general@gump.apache.org
Subject: Parallelism (was Re: A Few Plans)
References: <874oh0lpta.fsf@v35516.1blu.de>
	<F6F57C49-7667-4990-BF38-50180E829C24@apache.org>
Date: Mon, 21 Jun 2010 06:34:07 +0200
In-Reply-To: <F6F57C49-7667-4990-BF38-50180E829C24@apache.org> (Sander Temme's
	message of "Sun, 20 Jun 2010 21:44:18 +0200")
Message-ID: <87631d9f7k.fsf_-_@v35516.1blu.de>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

On 2010-06-20, Sander Temme wrote:

> On Jun 18, 2010, at 10:15 AM, Stefan Bodewig wrote:

>> Hi,

>> I just wanted to share a few plans I have short/midterm.  Feel free to
>> comment, pick tasks or add wishes.

>> Honestly I have no idea how we could deal with the ever increasing build
>> times as Gump grows, apart from some sort of distributed Gump which I
>> wouldn't want to build on top of the current code base (I'd rather think
>> in a tuple spaces architecture like Mnesia and Erlang or JavaSpaces and
>> anything on the VM).

> When I come back from vacation, we'll be pressing into service a dual
> quad core Apple Xserve with 6Gb of memory.

Sounds great - enjoy your vacation.

> This would allow for a little more concurrency than we can do on the
> Zone or VM... of course we'd have to be able to address all of those
> cores.  Wonder whether Python has glue for Grand Central Dispatch?

Right now Gump is a controlling process that spins of new processes, so
doing more in parallel would mean run those processes in parallel -
nothing Python would need to support.

If you log into one of the machines while Gump is running, the system
feels sluggish and any opration that hits the file system takes ages
which makes me fear we are I/O bound rather than CPU bound - making
those cores do more may not help too much in that case.  I can certainly
be wrong.

IIRC Gump's trunk supports parallel SCM checkouts but we've restricted
it to a maximum of one updater because Adam saw problems - it's been a
long time.

Currently we don't support building things in parallel at all.  Starting
several Ant or make builds in parallel would likely do what you expect,
but I don't know how mvn would deal with multiple processes accessing
the same local repository (and writing to it) in parallel.

It may be possible to construct concurrency in a way that is more or
less safe so that long running and self-contained builds like test-ant
could be spun off but all mvn builds that accessed the same local repo
would get serialized.  Of course it would take somebody to write the
code 8-)

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org