continuum-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Blevins <david.blev...@visi.com>
Subject Re: gbuild: dream to reality
Date Wed, 23 Nov 2005 23:41:52 GMT
Ok, here are some of the details I promised.  Cc'ing Continuum as as  
much of this code as possible is destined to them.  The idea is that  
we only should need to maintain a few extensions (which I spelled as  
extention in the code, oops) for the things that are specific to us  
or they don't want.

BASIC PARTS
--------------------------------------
Executing Agent (runs builds)
Producing Agent (creates builds)
Results Agent (aggregates build results)

Each agent consumes or produces on a queue or topic (or both).  You  
can have as many of each as you need or what.  They can run in the  
same process or separate processes.
All messages are Maps that are serialized to ObjectMessages onto  
either the BUILD.TASKS queue or the BUILD.RESULTS topic.

Excecuting Agent
-------------------
Runs continuums various continuum componentry and "update-from-scm"  
and "execute-builder" actions.  When this is done, the code has been  
updated and built.  Has extendible pre and post checking of incoming  
builds and outgoing build results.  Specifically there is a  
BuildAgentExtension (when spelling is fixed) that has a pre(Map  
incoming) method and a post(Map incoming, Map outgoing) method.

   Implemented Extensions so far:
   - FileIncludeExtension:  Checks the build request for a  
configurable header format.  Any headers matching that format are  
assumed to point to files that should be read in and included as text  
in the response.  The file must be one under the project's work  
directory.
   - HeaderIncludeExtension:  Any headers that match the configured  
pattern are copied as-is from the build request to the response message.

Producing Agent
-------------------
Scans a directory for properties files that define builds to add to  
the build queue.  This part could use some work.  The implementation  
that uses properties files is fine for shoving lots of work onto a  
queue in a really easy way, but this is where we also need some more  
integration with continuum as a frontend.  An implementation of A  
Producing Agent could be an alternate implementation of the Continuum  
build queue that just turns the build tasks into items that go onto  
the jms build queue.  Once you do something like that, you need to  
consider how things work in Continuum in regards to it's single  
threaded nature that seems to be assumed in places of the code.

Results Agent
-------------------
Very bare bones agent that consumes from teh BUILD.RESULTS topic and  
invokes BuildResultsExtensions (when spelling is fixed).  Performs no  
other work aside from invoking the extensions.  We can really go nuts  
with extensions here.  No limit to the ones we can creat to log,  
track, compile, report, analyze or otherwise process the results data  
in various ways.

The only BuildResultsExtension so far is one that scans for entries  
in the results map matching a regex ("^include.*" is the standard).   
It takes the value of that entry and writes it into a file.  The name  
of the file can be created using entries in the map as well.  So for  
example:
   {project.id}/{build.name}-{date}.properties



Some things we could add or do
--------------------------------

  - System status topic and agent
  - A page to display system status (the status of machines online)
  - Notification topic and agent that delegates to the notification  
manager
  - Something to browse the queue and displays the contents
  - A results extension to record how long each build took (data  
already available)
  - Something to use the average time for a build definition and the  
work in the queue and estimates the total hours of work to be done  
the divides it by the number of machines online and estimates a  
completion time.
  - Make client connection a component so it can easily be shared  
between agents
  - Security: maybe ssh-keys for all agents.  use SignedObject and  
SealedObject for message integrity and host verification.
  - Sort properties files when creating build definitions.
  - Have DirectoryMonitor file updated events add definitions to the  
queue.  Allows for builds to be in tasks dir and us to just exec  
'touch foo.properties' to queue up the builds inside.
  - Add gbuild agent version to message headers



That's all I have time for now.  Hope this can get some conversations  
started on where to go from here.

-David

On Nov 22, 2005, at 2:44 AM, David Blevins wrote:

> On Oct 26, 2005, at 2:01 AM, David Blevins wrote:
>> When I close my eyes and think big, I see a large federation  
>> consisting of smaller groups of machines from individuals and  
>> companies sharing some common building/testing infrastructure,  
>> open to and co-maintained by members of the community projects,  
>> building all our code all the time and testing it on every variety  
>> of OS, VM and Database imaginable....
>
> I'm happy to say...I so nailed this.  We've got it, it works, it  
> rocks.  It's running right now!
>
> For those who haven't been following the gbuild commit logs over  
> the last two weeks, I've taken Continuum, ActiveMQ, and a whole lot  
> of long nights and days and created a system that can push  
> continuum builds onto a jms queue (activemq) where they are picked  
> up by GBuild Agents (slimmed and extended version of continuum)  
> running in the gbuild network. The results are then pushed onto a  
> topic where they are aggregated by another agent in the gbuild  
> network.  Putting more work on the queue is as easy as dropping a  
> properties file in a directory.
>
> At the moment, have the following machines chugging away on a mix  
> of tck and other work:
>
>  - stan.gbuild.org
>  - kyle.gbuild.org
>  - kenny.gbuild.org
>  - cartman.build.org
>
> Going to get the Chariot Solutions boxes chef.gbuild.org,  
> timmy.gbuild.org, and jimmy.gbuild.org running tomorrow.
>
> A really cool aspect of this is that we can throw more machines at  
> it in the middle of a run.  It's just jms queue/consumer underneath  
> so new consumers can be added at anytime!  The whole thing is also  
> really extendible and not specific to tck testing, so we could work  
> in performance testing and whip up results agents to analyze the  
> performance data.  Sky is the limit!
>
> Off to get some sleep finally... Will send more technical  
> information tomorrow.  Just wanted to give everyone the good news.
>
> Thanks again to everyone (Simula, Mergere, Chariot, Dain) for  
> donating hosting, boxes, and manpower to get GBuild going.  Can't  
> wait to get the AMD boxes in the mix too.  I'm really excited about  
> all this and can't imagine it going any better.
>
> -David
>
>


Mime
View raw message