geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Mulder <>
Subject Re: gbuild: dream to reality
Date Thu, 24 Nov 2005 02:55:16 GMT
Sounds very cool -- congrats on getting this all together!


On 11/23/05, David Blevins <> wrote:
> Ok, here are some of the details I promised.  Cc'ing Continuum as as
> much of this code as possible is destined to them.  The idea is that
> we only should need to maintain a few extensions (which I spelled as
> extention in the code, oops) for the things that are specific to us
> or they don't want.
> --------------------------------------
> Executing Agent (runs builds)
> Producing Agent (creates builds)
> Results Agent (aggregates build results)
> Each agent consumes or produces on a queue or topic (or both).  You
> can have as many of each as you need or what.  They can run in the
> same process or separate processes.
> All messages are Maps that are serialized to ObjectMessages onto
> either the BUILD.TASKS queue or the BUILD.RESULTS topic.
> Excecuting Agent
> -------------------
> Runs continuums various continuum componentry and "update-from-scm"
> and "execute-builder" actions.  When this is done, the code has been
> updated and built.  Has extendible pre and post checking of incoming
> builds and outgoing build results.  Specifically there is a
> BuildAgentExtension (when spelling is fixed) that has a pre(Map
> incoming) method and a post(Map incoming, Map outgoing) method.
>    Implemented Extensions so far:
>    - FileIncludeExtension:  Checks the build request for a
> configurable header format.  Any headers matching that format are
> assumed to point to files that should be read in and included as text
> in the response.  The file must be one under the project's work
> directory.
>    - HeaderIncludeExtension:  Any headers that match the configured
> pattern are copied as-is from the build request to the response message.
> Producing Agent
> -------------------
> Scans a directory for properties files that define builds to add to
> the build queue.  This part could use some work.  The implementation
> that uses properties files is fine for shoving lots of work onto a
> queue in a really easy way, but this is where we also need some more
> integration with continuum as a frontend.  An implementation of A
> Producing Agent could be an alternate implementation of the Continuum
> build queue that just turns the build tasks into items that go onto
> the jms build queue.  Once you do something like that, you need to
> consider how things work in Continuum in regards to it's single
> threaded nature that seems to be assumed in places of the code.
> Results Agent
> -------------------
> Very bare bones agent that consumes from teh BUILD.RESULTS topic and
> invokes BuildResultsExtensions (when spelling is fixed).  Performs no
> other work aside from invoking the extensions.  We can really go nuts
> with extensions here.  No limit to the ones we can creat to log,
> track, compile, report, analyze or otherwise process the results data
> in various ways.
> The only BuildResultsExtension so far is one that scans for entries
> in the results map matching a regex ("^include.*" is the standard).
> It takes the value of that entry and writes it into a file.  The name
> of the file can be created using entries in the map as well.  So for
> example:
>    {}/{}-{date}.properties
> Some things we could add or do
> --------------------------------
>   - System status topic and agent
>   - A page to display system status (the status of machines online)
>   - Notification topic and agent that delegates to the notification
> manager
>   - Something to browse the queue and displays the contents
>   - A results extension to record how long each build took (data
> already available)
>   - Something to use the average time for a build definition and the
> work in the queue and estimates the total hours of work to be done
> the divides it by the number of machines online and estimates a
> completion time.
>   - Make client connection a component so it can easily be shared
> between agents
>   - Security: maybe ssh-keys for all agents.  use SignedObject and
> SealedObject for message integrity and host verification.
>   - Sort properties files when creating build definitions.
>   - Have DirectoryMonitor file updated events add definitions to the
> queue.  Allows for builds to be in tasks dir and us to just exec
> 'touch' to queue up the builds inside.
>   - Add gbuild agent version to message headers
> That's all I have time for now.  Hope this can get some conversations
> started on where to go from here.
> -David
> On Nov 22, 2005, at 2:44 AM, David Blevins wrote:
> > On Oct 26, 2005, at 2:01 AM, David Blevins wrote:
> >> When I close my eyes and think big, I see a large federation
> >> consisting of smaller groups of machines from individuals and
> >> companies sharing some common building/testing infrastructure,
> >> open to and co-maintained by members of the community projects,
> >> building all our code all the time and testing it on every variety
> >> of OS, VM and Database imaginable....
> >
> > I'm happy to say...I so nailed this.  We've got it, it works, it
> > rocks.  It's running right now!
> >
> > For those who haven't been following the gbuild commit logs over
> > the last two weeks, I've taken Continuum, ActiveMQ, and a whole lot
> > of long nights and days and created a system that can push
> > continuum builds onto a jms queue (activemq) where they are picked
> > up by GBuild Agents (slimmed and extended version of continuum)
> > running in the gbuild network. The results are then pushed onto a
> > topic where they are aggregated by another agent in the gbuild
> > network.  Putting more work on the queue is as easy as dropping a
> > properties file in a directory.
> >
> > At the moment, have the following machines chugging away on a mix
> > of tck and other work:
> >
> >  -
> >  -
> >  -
> >  -
> >
> > Going to get the Chariot Solutions boxes,
> >, and running tomorrow.
> >
> > A really cool aspect of this is that we can throw more machines at
> > it in the middle of a run.  It's just jms queue/consumer underneath
> > so new consumers can be added at anytime!  The whole thing is also
> > really extendible and not specific to tck testing, so we could work
> > in performance testing and whip up results agents to analyze the
> > performance data.  Sky is the limit!
> >
> > Off to get some sleep finally... Will send more technical
> > information tomorrow.  Just wanted to give everyone the good news.
> >
> > Thanks again to everyone (Simula, Mergere, Chariot, Dain) for
> > donating hosting, boxes, and manpower to get GBuild going.  Can't
> > wait to get the AMD boxes in the mix too.  I'm really excited about
> > all this and can't imagine it going any better.
> >
> > -David
> >
> >

View raw message