avalon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Simons <leosim...@apache.org>
Subject Re: [GUMP@lsd]: avalon/avalon failed
Date Fri, 27 Feb 2004 21:12:31 GMT
Niclas Hedhman wrote:
> On Friday 27 February 2004 13:51, Gump Integration Build wrote:
> 
>>To whom it may engage...
> 
> Leo, Having looked at Gump a couple of times, I am still confused over how it 
> works and basically have no clue.

:D

In short, gump is a python program run from a crontab each night. It 
reads a whole slew of XML descriptors that describe how to build a whole 
slew of projects from CVS, making sure that the classpaths used in those 
builds contain the stuff it has just built. It keeps track of success, 
failure, dependency failure, dependee failure, etc. It generates 
webpages from those logs, and sends out a set of e-mails to mailing 
lists identified in those XML descriptors.

> 1. Why doesn't it pick up Maven project descriptors instead

gump predates maven. It should have had this functionality bu now, but 
there has been (IMNSHO) insufficient interest in the maven developer 
community to help the gump team make that possible.

>, since to me it 
> seems that most is available in that? Does it have to use Ant? 

gump its descriptor format was around before maven, and it assumes the 
use of ant. Work is ongoing by a few peeps to make gump play nice with 
maven, but that's not exactly trivial (the goal for gump is to build the 
latest of everything, including the latest of maven, and maven is not 
easy to bootstrap).

volunteers welcome ;)

> 2. Can you give some pointer which explains the rationale instead of the 
> details, and "how to go about" getting it to work?

maybe. Let me rant a little. Vague question gets a long answer ;)

gump is about "continuous integration". All the open source java 
projects out there are highly dependent on each other, and a change in 
some of them (like ant or xerces avalon-framework or logkit, you know, 
the common dependency set for lots of projects) can causes lots of ripples.

The goal is to grab these early on. So gump tries to emulate a kind of 
'fanatic' developer who grabs all the latest from CVS each night, 
figures out the dependency trees of all the projects, starts at the 
bottom (bootstrapping ant), and then works its way all up into the far 
corners of the tree. Along the way, gump notes what works and what 
doesn't, does classpath management, etc etc.

Since gump is automated, it needs to be told what each and every project 
needs (dependencies), any kind of custom settings for the integration 
build, and how to invoke the relevant tools (cvs, ant), like what ant 
target to run in what directory. This is done through XML descriptors.

Gump doesn't know how to run maven, so for maven projects we have a 
special ant buildfile which emulates maven its behaviour.

Going about getting it to work is a bit of an error-prone process. If 
something fails, change a little piece of the puzzle, and see if the 
next gump run succeeds a little better. It's like magic. The fact that 
it is like magic is an indication of just how bad our build system is!

Gump integration for avalon is difficult, but also vital. 103 projects 
that participate in gump builds depend on avalon-framework.jar, and if 
it doesn't get build its not possible to detect what problems changes in 
any of those projects causes to another.

Since we have a complex everchanging build system with complex 
everchanging use of maven (and maven is complex and everchanging 
itself), our special ant buildfile is complex as well, and our 
descriptors are everchanging as well (though not particularly complex).

Getting things to work involves synchronizing our ant emulator with the 
actual maven build, synchronizing the gump descriptor with the inputs 
and outputs of the build process, and especially figuring out what to 
change where.

With the current issue, someone changed something somewhere in some 
piece of our build system that caused avalon-framework to stop building 
in gump.

To figure out what is going on, you need to take a look at the output 
gump puts on the web each night. It's pretty specific about what its doing:

http://lsd.student.utwente.nl/gump/avalon/gump_work/build_avalon_avalon.html

this error indicates some kind of classpath conflict. I don't understand 
it yet. What's making matters worse, but can also be a clue, is that an 
older version of gump doesn't have the same problem:

http://gump.covalent.net/log/avalon.html

likely because it contains some dirty hack that isn't in the new one :D

> For instance, in the nag email that is coming, there are references to 
> non-existent directories. Is that a declaration in Avalon Gump descriptors or 
> is part of something else?

these references are in the descriptor (the <work/> tag):

http://lsd.student.utwente.nl/gump/avalon/avalon_details.html#Definition

the problem seems to be that the descriptor is out of sync with what 
actually happens on invoking our ant script (the maven-common-gump-build 
file).

The only way to improve our "Friend of Gump" factor (which is a rough 
statistic measure of exactly how much "pain" we cause our dependencies; 
http://lsd.student.utwente.nl/gump/avalon/avalon.html#Statistics), is by 
having each and every developer who commits changes monitor what their 
effects is during gump's integration run.

Another helpful measure is keeping the gump descriptors in the gump cvs, 
where every ASF committer can freely change them. The downside to this 
is that people on the gump team will just silently work on keeping our 
descriptors in sync with our build process and the avalon developers 
will not bother with it, while it should really be our responsibility. 
Avalon has historically not been a very good "friend of gump".

I'm not actually very intimately familiar with gump internals; if you 
have more detailed questions (or want absolutely factually correct 
answers :-D), it would be good to (cross)post them to the gump list. I 
can also get you an account on my box (which is currently one of the 
main gump servers) if you want to take a look around.

-- 
cheers,

- Leo Simons

-----------------------------------------------------------------------
Weblog              -- http://leosimons.com/
IoC Component Glue  -- http://jicarilla.org/
Articles & Opinions -- http://articles.leosimons.com/
-----------------------------------------------------------------------
"We started off trying to set up a small anarchist community, but
  people wouldn't obey the rules."
                                                         -- Alan Bennett



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
For additional commands, e-mail: dev-help@avalon.apache.org


Mime
View raw message