gump-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject svn commit: r123431 - in gump/dynagump/trunk/webapp: . docs docs/howto
Date Mon, 27 Dec 2004 16:58:38 GMT
Author: leosimons
Date: Mon Dec 27 08:58:37 2004
New Revision: 123431

Work on new documentation. Still a lot to do...
      - copied, changed from r123404, gump/dynagump/trunk/webapp/docs/index.xml
      - copied, changed from r123404, gump/dynagump/trunk/webapp/docs/index.xml
   gump/dynagump/trunk/webapp/docs/process.png   (contents, props changed)

Copied: gump/dynagump/trunk/webapp/docs/HowGumpWorks.xml (from r123404, gump/dynagump/trunk/webapp/docs/index.xml)
--- gump/dynagump/trunk/webapp/docs/index.xml	(original)
+++ gump/dynagump/trunk/webapp/docs/HowGumpWorks.xml	Mon Dec 27 08:58:37 2004
@@ -1,30 +1,99 @@
-   <title>Gump | Documentation</title>
+   <title>Gump | Documentation | How Gump Works</title>
     <item name="Home" url="../"/>
-    <current name="Documentation"/>
+    <item name="Documentation" url="./"/>
+    <current name="How Gump Works"/>
     <item name="Results" url="../results/"/>
-    <current name="Documentation"/>
+    <current name="Documentation" url="./"/>
+    <item name="Wiki" url=""/>
     <item name="Home" url="../"/>
-   <sidebar>
-    <item name="Philosophy" url="">
-     <description>Here is where you find out more about the gump philosophy and why
it was built.</description>
-    </item>
-    <item name="Metadata" url="">
-     <description>Here is where you find out more about the gump metadata descriptors.</description>
-    </item>
-   </sidebar>
-     <h1>Documentation</h1>
-     <p>Just a placeholder for now but hopefully it will grow.</p>
+<h1>How Gump Works</h1>
+<p>Please note that this documentation specifies the plans we have,
+rather than what we actually do right now. The key difference is that
+Gump will be made much more intelligent about whom to send what e-mail
+and what to put in those e-mails. However, the basic ideas for all this
+are quickly falling into place.</p>
+<h2>The actual process</h2>
+<img src="process.png" width="300" align="left"/>
+<p>Lets start with the low-level overview. Gump has two main parts. The
+first part takes care of actually performing the build. Internally, we
+tend to call this part "Gumpy", since its written in python. The second
+part takes care of the reporting and introspection. It is written in
+java using Apache Cocoon.</p>
+<h3>Gumpy, the builder</h3>
+<p>Gumpy is a python script that is fired from cron every few hours. It
+does some self-initialization, looks at it environment, then starts
+figuring out what to do. It updates the gump CVS and SVN repositories.
+The next task is to read in all gump project definitions. For
+maven-based projects, we read in maven's project.xml files and do some
+processing on those. For projects using other build tools (like Ant),
+an XML descriptor describing that project is kept in the Gump
+subversion repository.</p>
+<p>All these project definitions are merged into a database, which we
+the GumpModelDatabase. Once our database is filled, we start walking it
+to figure out the dependencies between all the projects. We then figure
+out a build order. With our model serialized, we know exactly what to
+<p>In our just-determined order, we run CVS or SVN updates for our
+projects, then try to build them (using ant or maven). The precise
+build commands to use as well as any environment settings (such as
+references to installed packages like javamail) are all figured out
+from the gump project definitions (see HowToGetBuiltByGump for more
+information on how that works).</p>
+<p>Now, the tricky bit is specifying dependencies. Rather than use the
+maven repository or some other place where jars are stored, we set the
+system classpath to include all the dependencies declared for a
+project. And rather than point to a specific versioned jar or library,
+we opt to use the output of the projects that we just built straight
+from bleeding edge SVN.</p>
+<p>Gumpy keeps track of everything it does, and what the results of
+actions are. Success and failure states, build logs, and much more are
+all saved and stored into another database, which we call the
+ResultMetadataDatabase. This database doesn't just store the current
+run, but maintains information for all previous runs as well.</p>
+<p>If a project fails to build, Gumpy will take special actions to try
+figure out what caused that failure. The algorithm used to do this is
+based on a bit of graph theory, which won't be described here.
+Interested people should look at the GumpAlgorithm page. (NOTE: this
+algorithm has been written but not yet implemented!) Suffice saying
+that Gumpy is often able to determine exactly which CVS or SVN commit
+caused the build failure, and in other instances is able to give a good
+<h3>DynaGump, the reporter</h3>
+<p>Once Gumpy is done, its time for DynaGump to do its thing. DynaGump
+a web application which hooks into the ResultMetadataDatabase to
+generate reports. These reports allow anyone to find out how stable the
+build for a particular project is, what the main issues are, and the
+like. We produce figures describing the build history of a project.
+This can help developers figuring out which dependencies are causing
+them problems, but also how their changes affect the projects that
+depend on them (their dependees).</p>
+<p>If there's an issue, DynaGump figures out who needs to know about
+and sends them e-mail. For example, if a project introduced a code
+change that results in one or more of their dependees failing to build,
+we'll let that project know, as well as the project that broke. This
+allows all the developers involved to start collaborating immediately
+to resolve the issue.</p>
+<p>Advanced reporting and workflow features (similar to the Jira
+Dashboard) feature will be made available. For example, you will be
+able to create a custom page which lists information about all the
+projects that you are interested in, or get that information as an RSS

Copied: gump/dynagump/trunk/webapp/docs/Philosophy.xml (from r123404, gump/dynagump/trunk/webapp/docs/index.xml)
--- gump/dynagump/trunk/webapp/docs/index.xml	(original)
+++ gump/dynagump/trunk/webapp/docs/Philosophy.xml	Mon Dec 27 08:58:37 2004
@@ -1,30 +1,29 @@
-   <title>Gump | Documentation</title>
+   <title>Gump | Documentation | Philosophy</title>
     <item name="Home" url="../"/>
-    <current name="Documentation"/>
+    <item name="Documentation" url="./"/>
+    <current name="Philosophy"/>
     <item name="Results" url="../results/"/>
-    <current name="Documentation"/>
+    <current name="Documentation" url="./"/>
     <item name="Home" url="../"/>
-   <sidebar>
-    <item name="Philosophy" url="">
-     <description>Here is where you find out more about the gump philosophy and why
it was built.</description>
-    </item>
-    <item name="Metadata" url="">
-     <description>Here is where you find out more about the gump metadata descriptors.</description>
-    </item>
-   </sidebar>
-     <h1>Documentation</h1>
-     <p>Just a placeholder for now but hopefully it will grow.</p>
-   </content>
+<h1>Gump Philosophy</h1>
    I've read with great amusement the motivations attributed to why Gump
originally written.  The reasons were most definitely not altruistic.
    To understand why
Gump was written, you need to understand a bit of
    the history of the Jakarta and XML projects.
 But, first, here is a
    concise and concrete summary:

is easier to get a patch accepted against the most current version
    of a project than some
previous baseline.</li>
<li>It is much more effective to express your opinion
on a change that
    will affect you <strong>before</strong> that change is released
than afterwards.</li>


<p>To help set the context, in 1999 the ink was hardly dry
on the XML 
    specifications; and what is now known as XSLT, FOP, and XPath were all 
  interwoven in a confused jumble.  JDK 1.2 was only recently introduced
    and wouldn't
be widely adopted for many years (to this day, a number
    of projects retain JDK 1.1 compatibility).</p>
this harsh environment, it is amazing that open source software grew
    at all.  In order
to make progress, early versions of Xalan required
    specific versions of Xerces.  Cocoon
required specific versions
    of both.  Cocoon also depended on other projects which were
two years
    away from having alpha releases - Turbine and Avalon.</p>
virtually every project required their own specific version of
found this time very frustrating.  As an active developer at the 
    time on projects such
as Ant and Tomcat, I wanted to make improvements
    to these projects; but was precluded
from using the fruits of my own
    labors when working with projects like cocoon.  Ultimately,
I found
    that I could get what I wanted by writing my own scripts to set up
    the classpath
as I wanted.</p>
<p>In 2000, Ant developed the ability to control one's classpath.
   This completely defeated my ability to select which versions I wanted
    to use.  At first,
this was done on projects I wasn't active in, but
    eventually it encroached everywhere.</p>
one day a change was checked in that completely broke me.  As
    an active committer on the
project, I "-1'ed" the change.  That -1 was
    declared invalid, shouted down, and ignored.</p>
I did the only thing I could think of.  I hacked Ant so that I 
    could tell it to ignore
the classpath elements in build scripts.  All of 
    them.  This was then and remains now
a dirty, nasty hack.  It meant that 
    test cases could not be run using the recently compiled
classes unless I 
    knew in advance where the classes were to be placed.  It means that
    can't be built in one run against both Tomcat 3 and Tomcat 4.</p>
it was the only way I could regain control.</p>

that I had a technological solution to my problems, the number of
    projects I was interested
in following grew.  This resulted in a number
    of problems of scalability: the first being
that the scripts were 
    becoming more difficult to maintain, and second that the long chains
    dependencies were fragile and only worked if the participants were 
in supporting the combination of projects that I wanted to 
solution to my problems was to automate the generation of these
    scripts and to publish
the results.</p>
<p>The results exceeded my expectations.  People who claimed
that version
    incompatibility problems were rare and easily addressed could now see
 that it was a daily event - in fact to this day, I have only seen a
    clean run a mere
handful of times.  But what I could provide to developers
    was timely feedback on the impact
of their changes on others, and in turn 
    help them make their case against changes that
would affect them.</p>
<p>Most telling of all were the changes to three projects:
Ant, Turbine,
    and Avalon.  All three were notorious for making changes which broke
 their users.  In the cases of Turbine and Avalon, these projects were
    perennially in
alpha, so it was argued that nobody should depend on them.
    Now, there are stable releases
of each.  Deprecation is now actively 
    used by all three.</p>

<p>There is a growing trend for projects to split into multiple
cvs trees -
    Avalon, Tomcat, and Turbine are examples.  The creation of "commons"
shared between projects makes it more and more likely that other
    developers are interested
in multiple code bases and face some of the same
    problems that I have faced.</p>
these reasons, I'd like to see Gump evolve into an developer tool.  
    The overall concept
is as follows: one describes using simple XML 
    constructs the combination of projects
and versions thereof that one 
    desires in their workspace, and customized scripts are
built to order.  
    The results of one's builds are conveniently published as HTML.</p>
even with a GUI.</p>
<p>With Ant becoming stable and backwards compatible, I'd
also like to
    see a movement towards it becoming something that you install and
upgrade, as opposed to something that you download with
    each and every project.</p>
that XML parsers and XSL translators are pluggable and default
    versions will be shipping
with the JDK, it would also be nice if each
    project stopped shipping duplicate copies
of these jars too, and instead
    used the one of your choosing.</p>
+<h2 class="boxed">2003</h2>
<p>As a heavy user of OSS software, combining
lots of it into deep stacks,
  		<a href="">JAR
Hell</a> became
  		a pet peave of <a href="">Adam
  		Supporting heathly re-use became a passion.
keeps software re-use fresh and healthy and reduces JAR Hell.</p>

Added: gump/dynagump/trunk/webapp/docs/howto/index.xml
--- (empty file)
+++ gump/dynagump/trunk/webapp/docs/howto/index.xml	Mon Dec 27 08:58:37 2004
@@ -0,0 +1,31 @@
+   <title>Gump | Documentation | How-To's</title>
+   <path>
+    <item name="Home" url="../../"/>
+    <item name="Documentation" url="../"/>
+    <current name="How-To's"/>
+   </path>
+   <tabs>
+    <item name="Results" url="../../results/"/>
+    <current name="Documentation" url="../"/>
+    <item name="Wiki" url=""/>
+    <item name="Home" url="../../"/>
+   </tabs>
+   <content>
+     <h1>How-To Documentation</h1>
+<p>Concrete guides to working with or on gump:</p>
+  <li><a href="HowToFixGumpBuildProblems.html">How To Fix Gump Build
+Problems</a>: this document explains the steps you should take if gump send you or
+a project you participated in an e-mail.</li>
+  <li><a href="HowToGetBuiltByGump.html">How To Get Built By Gump</a>:
this document explains how to add new projects to gump, focussing
+mostly on the gump object model, our xml descriptors that tell gump how
+to "do its thing".</li>
+   </content>

Modified: gump/dynagump/trunk/webapp/docs/index.xml
--- gump/dynagump/trunk/webapp/docs/index.xml	(original)
+++ gump/dynagump/trunk/webapp/docs/index.xml	Mon Dec 27 08:58:37 2004
@@ -10,21 +10,39 @@
     <item name="Results" url="../results/"/>
     <current name="Documentation"/>
+    <item name="Wiki" url=""/>
     <item name="Home" url="../"/>
-    <item name="Philosophy" url="">
-     <description>Here is where you find out more about the gump philosophy and why
it was built.</description>
+    <item name="Philosophy" url="Philosophy.html">
+     <description>Find out why gump was built and what our goals are with the gump
-    <item name="Metadata" url="">
-     <description>Here is where you find out more about the gump metadata descriptors.</description>
+    <item name="How Gump Works" url="HowGumpWorks.html">
+     <description>Get the 10-mile-high overview of the gump architecture and workflow.</description>
+    </item>
+    <item name="How-To's" url="howto/">
+     <description>Get concrete guides to working with and on gump.</description>
+    </item>
+    <item name="Metadata" url="">
+     <description>Look up the details of the gump descriptor syntax.</description>
-     <p>Just a placeholder for now but hopefully it will grow.</p>
+<p>Gump is not a trivial project. We compile thousands of sourcefiles
+every night, and try to send the right reports to hundreds of projects
+when something bad happens. This means that there's a lot to "get"
+about gump. Besides the information on this part of our website, make
+sure to take a look at:</p>
+  <li>the <a href="">Main gump website</a> -- you're
+looking at our new documentation (ie, a work in progress). The main website
+contains a wealth of other information.</li>
+  <li>the <a href="">Gump wiki</a> -- Gump
has an
+active wiki with a lot of tidbits of information.</li>

Added: gump/dynagump/trunk/webapp/docs/process.png
Binary file. No diff available.

Modified: gump/dynagump/trunk/webapp/docs/sitemap.xmap
--- gump/dynagump/trunk/webapp/docs/sitemap.xmap	(original)
+++ gump/dynagump/trunk/webapp/docs/sitemap.xmap	Mon Dec 27 08:58:37 2004
@@ -8,8 +8,36 @@
+   <!-- index -->
    <map:match pattern="">
     <map:generate src="index.xml"/>
+    <map:transform src="../stylesheets/page2html.xslt"/>
+    <map:serialize type="xhtml"/>
+   </map:match>
+   <!-- index -->
+   <map:match pattern="**/">
+    <map:generate src="{1}/index.xml"/>
+    <map:transform src="../stylesheets/page2html.xslt"/>
+    <map:serialize type="xhtml"/>
+   </map:match>
+    <!-- images -->
+    <map:match pattern="**.gif">
+      <map:read mime-type="images/gif" src="{1}.gif"/>
+    </map:match>
+    <map:match pattern="**.png">
+      <map:read mime-type="images/png" src="{1}.png"/>
+    </map:match>
+    <map:match pattern="**.jpg">
+      <map:read mime-type="images/jpeg" src="{1}.jpg"/>
+    </map:match>
+   <!-- docs -->
+   <map:match pattern="**.html">
+    <map:generate src="{1}.xml"/>
     <map:transform src="../stylesheets/page2html.xslt"/>
     <map:serialize type="xhtml"/>

Modified: gump/dynagump/trunk/webapp/index.xml
--- gump/dynagump/trunk/webapp/index.xml	(original)
+++ gump/dynagump/trunk/webapp/index.xml	Mon Dec 27 08:58:37 2004
@@ -9,11 +9,39 @@
     <item name="Results" url="results/"/>
     <item name="Documentation" url="docs/"/>
+    <item name="Wiki" url=""/>
     <current name="Home"/>
-     <h1>Welcome to Apache Gump</h1>
+     <h1>Welcome to Apache Gump!</h1>
+<p>Gump is Apache's continuous integration tool. It is written in
+and fully supports Apache Ant, Apache Maven and other build tools. Gump
+is unique in that it builds and compiles software against the latest
+development versions of those projects. This allows gump to detect
+potentially incompatible changes to that software just a few hours
+after those changes are checked into the version control system.
+Notifications are sent to the project team as soon as such a change is
+detected, referencing more detailed reports available online.</p>
+<p>You can set up and run Gump on your own machine and run it on your
+projects, however it is currently most famous for building most of
+Apache's java-based projects and their dependencies (which constitutes
+several million lines of code split up into hundreds of projects). For
+this purpose, the gump project maintains its own dedicated server.
+<h1>Build results</h1>
+<p>Apache dedicates a complete machine to running gump. The
+<a href="results/">Build results</a> page provides a view of the data
+gump generates for apache.</p>
+<p>We're working on a <a href="docs/">new set of documentation</a>. A
+lot of material is still available on
+<a href="">our main website</a> only. Also make
+sure to take a look at the <a href="">wiki</a>.</p>

View raw message