geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Warner" <>
Subject Re: Continuous TCK Testing
Date Wed, 08 Oct 2008 16:05:37 GMT
Here's a quick question.  Where does AHP come from?

On Mon, Oct 6, 2008 at 1:18 PM, Jason Dillon <> wrote:

> Sure np, took me a while to get around to writing it too ;-)
> --jason
> On Oct 6, 2008, at 10:24 PM, Jason Warner wrote:
> Just got around to reading this.  Thanks for the brain dump, Jason.  No
> questions as of yet, but I'm sure I'll need a few more reads before I
> understand it all.
> On Thu, Oct 2, 2008 at 2:34 PM, Jason Dillon <>wrote:
>> On Oct 1, 2008, at 11:20 PM, Jason Warner wrote:
>>  Is the GBuild stuff in svn the same as the anthill-based code or is that
>>> something different?  GBuild seems to have scripts for running tck and that
>>> leads me to think they're the same thing, but I see no mention of anthill in
>>> the code.
>> The Anthill stuff is completely different than the GBuild stuff.  I
>> started out trying to get the TCK automated using GBuild, but decided that
>> the system lacked too many features to perform as I desired, and went ahead
>> with Anthill as it did pretty much everything, though had some stability
>> problems.
>> One of the main reasons why I choose Anthill (AHP, Anthill Pro that is)
>> was its build agent and code repository systems.  This allowed me to ensure
>> that each build used exactly the desired artifacts.  Another was the
>> configurable workflow, which allowed me to create a custom chain of events
>> to handle running builds on remote agents and control what data gets set to
>> them, what it will collect and what logic to execute once all distributed
>> work has been completed for a particular build.  And the kicker which help
>> facilitate bringing it all together was its concept of a build life.
>> At the time I could find *no other* build tool which could meet all of
>> these needs, and so I went with AHP instead of spending months
>> building/testing features in GBuild.
>> While AHP supports configuring a lot of stuff via its web-interface, I
>> found that it was very cumbersome, so I opted to write some glue, which was
>> stored in svn here:
>> Its been a while, so I have to refresh my memory on how this stuff
>> actually worked.  First let me explain about the code repository (what it
>> calls codestation) and why it was critical to the TCK testing IMO.  When we
>> use Maven normally, it pulls data from a set of external repositories, picks
>> up more repositories from the stuff it downloads and quickly we loose
>> control where stuff comes from.  After it pulls down all that stuff, it
>> churns though a build and spits out the stuff we care about, normally
>> stuffing them (via mvn install) into the local repository.
>> AHP supports by default tasks to publish artifacts (really just a set of
>> files controlled by an Ant-like include/exclude path) from a build agent
>> into Codestation, as well as tasks to resolve artifacts (ie. download them
>> from Codestation to the local working directory on the build agents system).
>>  Each top-level build in AHP gets assigned a new (empty) build life.
>>  Artifacts are always published to/resolved from a build life, either that
>> of the current build, or of a dependency build.
>> So what I did was I setup builds for Geronimo Server (the normal
>> server/trunk stuff), which did the normal mvn install thingy, but I always
>> gave it a custom -Dmaven.local.repository which resolved to something inside
>> the working directory for the running build.  The build was still online, so
>> it pulled down a bunch of stuff into an empty local repository (so it was a
>> clean build wrt the repository, as well as the source code, which was always
>> fetched for each new build).  Once the build had finished, I used the
>> artifact publisher task to push *all* of the stuff in the local repository
>> into Codestation, labled as something like "Maven repository artifacts" for
>> the current build life.
>> Then I setup another build for Apache Geronimo CTS Server (the
>> porting/branches/* stuff).  This build was dependent upon the "Maven
>> repository artifacts" of the Geronimo Server build, and I configured those
>> artifacts to get installed on the build agents system in the same directory
>> that I configured the CTS Server build to use for its local maven
>> repository.  So again the repo started out empty, then got populated with
>> all of the outputs from the normal G build, and then the cts-server build
>> was started.  The build of the components and assemblies is normally fairly
>> quick and aside from some stuff in the private tck repo won't download muck
>> more stuff, because it already had most of its dependencies installed via
>> the Codestation dependency resolution.   Once the build finished, I
>> published to cts-server assembly artifacts back to Codestation under like
>> "CTS Server Assemblies" or something.
>> Up until this point its normal builds, but now we have built the G server,
>> then built the CTS server (using the *exact* artifacts from the G server
>> build, even though each might have happened on a different build agent).
>>  And now we need to go and run a bunch of tests, using the *exact* CTS
>> server assemblies, produce some output, collect it, and once all of the
>> tests are done render some nice reports, etc.
>> AHP supports setting up builds which contain "parallel" tasks, each of
>> those tasks is then performed by a build agent, they have fancy build agent
>> selection stuff, but for my needs I had basically 2 groups, one group for
>> running the server builds, and then another for running the tests.  I only
>> set aside like 2 agents for builds and the rest for tests.  Oh, I forgot to
>> mention that I had 2 16x 16g AMD beasts all running CentOS 5, each with
>> about 10-12 Xen virtual machines running internally to run build agents.
>>  Each system also had a RAID-0 array setup over 4 disks to help reduce disk
>> io wait, which was as I found out the limiting factor when trying to run a
>> ton of builds that all checkout and download artifacts and such.
>> I helped the AHP team add a new feature which was an parallel iterator
>> task, so you define *one* task that internally fires off n parallel tasks,
>> which would set the iteration number, and leave it up to the build logic to
>> pick what to do based on that index.  The alternative was a unwieldy set of
>> like 200 tasks in their UI which simply didn't work at all.  You might have
>> notice an "iterations.xml" file in the tck-testsuite directory, this was was
>> was used to take an iteration number and turn it into what tests we actually
>> run.  The <iteration> bits are order sensitive in that file.
>> Soooo, after we have a CTS Server for a particular G Server build, we can
>> no go an do "runtests" for a specific set of tests (defined by an
>> iteration)... this differed from the other builds above a little, but still
>> pulled down artifacts, the CTS Server assemblies (only the assemblies and
>> the required bits to run the geronimo-maven-plugin, which was used to
>> geronimo:install, as well as used by the tck itself to fire up the server
>> and so on).  The key thing here, with regards to the maven configuration
>> (besides using that custom Codestation populated repository) was that the
>> builds were run *offline*.
>> After runtests completed, the results are then soaked up (the stuff that
>> javatest pukes out with icky details, as well as the full log files and
>> other stuff I can recall) and then pushed back into Codestation.
>> Once all of the iterations were finished, another task fires off which
>> generates a report.  It does this by downloading from Codestation all of the
>> runtests outputs (each was zipped I think), unzips them one by one, run some
>> custom goo I wrote (based some of the concepts from original stuff from the
>> GBuild-based TCK automation), and generates a nice Javadoc-like report that
>> includes all of the gory details.
>> I can't remember how long I spent working on this... too long (not the
>> reports I mean, the whole system).  But in the end I recall something like
>> running an entire TCK testsuite for a single server configuration (like
>> jetty) in about 4-6 hours... I sent mail to the list with the results, so if
>> you are curious what the real number is, instead of my guess, you can look
>> for it there.  But anyway it was damn quick running on just those 2
>> machines.  And I *knew* exactly that each of the distributed tests was
>> actually testing a known build that I could trace back to its artifacts and
>> then back to its SVN revision, without worrying about mvn downloading
>> something new when midnight rolled over or that a new G server or CTS server
>> build that might be in progress hasn't compromised the testing by polluting
>> the local repository.
>>  * * *
>> So, about the sandbox/build-support stuff...
>> First there is the 'harness' project, which is rather small, but contains
>> the basic stuff, like a version of ant and maven which all of these builds
>> would use, some other internal glue, a  fix for an evil Maven problem
>> causing erroneous build failures due to some internal thread state
>> corruption or gremlins, not sure which.  I kinda used this project to help
>> manage the software needed by normal builds, which is why Ant and Maven were
>> in there... ie. so I didn't have to go install it on each agent each time it
>> changed, just let the AHP system deal with it for me.
>> This was setup as a normal AHP project, built using its internal Ant
>> builder (though having that builder configured still to use the local
>> version it pulled from SVN to ensure it always works.
>> Each other build was setup to depend on the output artifacts from the
>> build harness build, using the latest in a range, like say using "3.*" for
>> the latest 3.x build (which looks like that was 3.7).  This let me work on
>> new stuff w/o breaking the current builds as I hacked things up.
>> So, in addition to all of the stuff I mentioned above wrt the G and CTS
>> builds, each also had this step which resolved the build harness artifacts
>> to that working directory, and the Maven builds were always run via the
>> version of Maven included from the harness.  But, AHP didn't actually run
>> that version of Maven directly, it used its internal Ant task to execute the
>> version of Ant from the harness *and* use the harness.xml buildfile.
>> The harness.xml stuff is some more goo which I wrote to help mange AHP
>> configurations.  With AHP (at that time, not sure if it has changed) you had
>> to do most everything via the web UI, which sucked, and it was hard to
>> refactor sets of projects and so on.  So I came up with a standard set of
>> tasks to execute for a project, then put all of the custom muck I needed
>> into what I called a _library_ and then had the AHP via harness.xml invoke
>> it with some configuration about what project it was and other build
>> details.
>> The actual harness.xml is not very big, it simply makes sure that */bin/*
>> is executable (codestation couldn't preserve execute bits), uses the
>> Codestation command-line client (invoking the javaclass directly though) to
>> ask the repository to resolve artifacts from the "Build Library" to the
>> local repository.  I had this artifact resolution separate from the normal
>> dependency (or harness) artifact resolution so that it was easier for me to
>> fix problems with the library while a huge set of TCK iterations were still
>> queued up to run.  Basically, if I noticed a problem due to a code or
>> configuration issue in an early build, I could fix it, and use the existing
>> builds to verify the fix, instead of wasting an hour (sometimes more
>> depending on networking problems accessing remote repos while building the
>> servers) to rebuild and start over.
>> This brings us to the 'libraries' project.  In general the idea of a
>> _library_ was just a named/versioned collection of files, where you could be
>> used by a project.  The main (er only) library defined in this SVN is
>> system/.  This is the groovy glue which made everything work.  This is where
>> the entry-point class is located (the guy who gets invoked via harness.xml
>> via:
>>    <target name="harness" depends="init">
>>        <groovy>
>>            <classpath>
>>                <pathelement location="${library.basedir}/groovy"/>
>>            </classpath>
>>            gbuild.system.BuildHarness.bootstrap(this)
>>        </groovy>
>>    </target>
>> I won't go into too much detail on this stuff now, take a look at it and
>> ask questions.  But, basically there is stuff in gbuild.system.* which is
>> harness support muck, and stuff in gbuild.config.* which contains
>> configuration.  I was kinda mid-refactoring of some things, starting to add
>> new features, not sure where I left off actually. But the key bits are in
>> gbuild.config.project.*  This contains a package for each project, with the
>> package name being the same as the AHP project (with " " -> "_"). And then
>> in each of those package is at least a Controller.groovy class (or other
>> classes if special muck was needed, like for the report generation in
>> Geronimo_CTS, etc).
>> The controller defines a set of actions, implemented as Groovy closures
>> bound to properties of the Controller class.  One of the properties passed
>> in from the AHP configuration (configured via the Web UI, passed to the
>> harness.xml build, and then on to the Groovy harness) was the name of the
>> _action_ to execute.  Most of that stuff should be fairly straightforward.
>> So after a build is started (maybe from a Web UI click, or SVN change
>> detection, or a TCK runtests iteration) the following happens (in simplified
>> terms):
>>  * Agent starts build
>>  * Agent cleans its working directory
>>  * Agent downloads the build harness
>>  * Agent downloads any dependencies
>>  * Agent invoke Ant on harness.xml passing in some details
>>  * Harness.xml downloads the system/1 library
>>  * Harness.xml runs gbuild.system.BuildHarness
>>  * BuildHarness tries to construct a Controller instance for the project
>>  * BuildHarness tries to find Controller action to execute
>>  * BuildHarness executes the Controller action
>>  * Agent publishes output artifacts
>>  * Agent completes build
>> A few extra notes on libraries, the JavaEE TCK requires a bunch of stuff
>> we get from Sun to execute.  This stuff isn't small, but is for the most
>> part read-only.  So I setup a location on each build agent where these files
>> were installed to.  I created AHP projects to manage them and treated them
>> like a special "library" one which tried really hard not to go fetch its
>> content unless the local content was out of date.  This helped speed up the
>> entire build process... cause that delete/download of all that muck really
>> slows down 20 agents running in parallel on 2 big machines with stripped
>> array.  For legal reasons this stuff was not kept in's
>> main repository, and for logistical reasons wasn't kept in the private tck
>> repo on either.  Because there were so many files, and be
>> case the httpd configuration on kicks out requests that it
>> thinks are *bunk* to help save the resources for the community, I had setup
>> a private ssl secured private svn repository on the old gbuild.orgmachines to put
in the full muck required, then setup some goo in the
>> harness to resolve them.  This goo is all in gbuild.system.library.*  See
>> the gbuild.config.projects.Geronimo_CTS.Controller for more of how it was
>> actually used.
>>  * * *
>> Okay, that is about all the brain-dump for TCK muck I have in me for
>> tonight.  Reply with questions if you have any.
>> Cheers,
>> --jason
> --
> ~Jason Warner

~Jason Warner

View raw message