ant-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Jackl-Mochel" <jmoc...@foliage.com>
Subject My experiences with doing a large project with ANT
Date Wed, 13 Mar 2002 20:26:48 GMT
Introduction
------------

Some people on this list have asked me to post details of how I got a
hierarchical ant based build system working for a project I am involved in.
This
is a simple attempt to get most of my experiences down in a way that might
be
useful to the ANT community at large. For brevity, I am intentionally
leaving
out most of the numerous false starts and blind paths I went down.  I am
detailing out the general rationales and notable blind alleys whenever I see
a chance to save someone else a great deal of time.

The project is fairly large: 60+ engineers distributed among three locations
in
the US. It is a J2EE project with multiple application servers and
webservers.
There were several projects under a single large umbrella project with some
code
dependencies among them.

At the point that I started the project I had a solid experience with
automated
build and testing in C/C++ and solid Java programming so I didn't figure
this
would be particularly tough. If anything, I thought that the flexibility and
introspective nature of java and java tools would make it even easier.

Requirements
------------

We had several requirements that made ANT seem like a godsend

0. We needed to support multiple version of the app server and several
different
compilation configurations (debug, release, and performance )

1. We wanted to do automated "overnight" builds that:

1.1 pulled down and labeled the source tree
1.2 compilation
1.3 Unit tested the standalone Java classes
1.4 Packaged the J2EE components for deployment
1.5 Deployed and tested the deployable components
1.6 Run suppoort tools like JTest, Doclint, and javadoc.
1.7 Archived the resulting deliverables
1.8 Mailed the results as necessary

3. We needed to support multiple projects under the same umbrella uber
project.

Basic History
-------------

We defined a directory structure and basic build structure and went to town.
We
were using Kawa as the IDE of choice. We were running a PERL script in a NT
"at"
command at various intervals to invoke ANT and send out the mail as
necessary.

Over time we ran into problems with build time scaling, revision control
tools
(speed and features), maintainability of the build files and getting
configuration information into the ANT build system in the first case.

The directory structure originally looked like this:

\<uber project>

    \build  - contains buildfiles (master.ant) and configuration files

    \builds
        \<appserverver>
            \<compilationconfig>
                \classes        - root of the compiled Java output
                \javadocs       - root of the Javadoc output
                \deployment     - For holding deployables like
EARs,WARs,JARs

    \<project1>
        \build                  - contains project build file project.ant
        \ears
            \<ear-name-1>
                \WEB-INF        - deployment descriptors

        \wars
            \<war-name-1>
                \WEB-INF        - deployment descriptors

        \development
            \<subproject-1>
                \build          - contains subproject buildfile
subproject.ant
                \src            - root of subprojects source tree
                \jars
                    \<jar-name-1>
                        \build  - contains jar.ant
                        \META-INF

                \ejb-jars
                    \<ejb-jar-name-1>
                        \build  - contains ejb-jar.ant
                        \META-INF

The directory structure involved a large number of design needs that I will
try
and summarize. We avoided a source directory off of the root \uber folder
because we had many projects and sub projects where the code was worked on
offsite, in isolation and we wanted to avoid needing to pull the whole tree
down
for someone working on a subset of the files. We used the ..\build folders
in an
attempt to enable automatic generation of ANT files and inclusions into ANT
files for hiearchical builds. We named the diferent levels of ant files
differently for ease in making sweeping changes to the different files. The
WEB-
INF and META-INF folders were allocated on a per ejb-jar or jar basis to
eliminate name collision or copying schemes during the build.

A total build went something like this:

ANT was invoked against \uber\build\master.ant.

It used ENTITY references to include basic templated targets and actions and
other configuration information.

master.ant then invoked the project.ant files in order for the different
targets
(such as compile, testclasses, assemble, package, deploy, testdeployables,
etc..
The project.ant files then invoked the subproject.ant files in the same way
and
they invoked the jar.ant and ejb-jar.ant files. Each of the subfiles
themselves
included the various rules, targets, and other information from the
\uber\build
directory.

The \uber\build\master.ant file looked like (in summary):
==========================================================

<?xml version="1.0"?>
<!-- Include the config file -->
<!DOCTYPE project [
    <!ENTITY Config SYSTEM "file:./config.ant">
    <!ENTITY PrepareTarget SYSTEM "file:./jar.prepare.target.ant">
]>

<project name="Master" default="compile">

    &Config;

	&PrepareTarget;

	<target name="clean" depends="prepare">
	 	<ant antfile="project.ant" dir="../ThirdParty/build" target="clean" />
	 	<ant antfile="project.ant" dir="../Reusable/build" target="clean" />
	 	<ant antfile="project.ant" dir="../Veracity/build" target="clean" />
	</target>

	<target name="compile" depends="prepare">
	 	<ant antfile="project.ant" dir="../ThirdParty/build" target="compile" />
	 	<ant antfile="project.ant" dir="../Reusable/build" target="compile" />
	 	<ant antfile="project.ant" dir="../Veracity/build" target="compile" />
	</target>

	<target name="unittestclasses" depends="compile,prepare">
	 	<ant antfile="project.ant" dir="../ThirdParty/build"
target="unittestclasses" />
	 	<ant antfile="project.ant" dir="../Reusable/build"
target="unittestclasses" />
	 	<ant antfile="project.ant" dir="../Veracity/build"
target="unittestclasses" />
	</target>

	<target name="assemble" depends="compile,prepare">
	 	<ant antfile="project.ant" dir="../ThirdParty/build" target="assemble" />
	 	<ant antfile="project.ant" dir="../Reusable/build" target="assemble" />
	 	<ant antfile="project.ant" dir="../Veracity/build" target="assemble" />
	</target>

</project>
==========================================================

The \uber\build\config.ant file looked like (in summary):
==========================================================
<!-- Properties coming in from the JVM -->
<!--
	We specifiy the paths to tools explicitly.

	This is done by taking in the named properties from the command line
->

<property name="ProjectHome" 		value="${PROJECTHOME}" />
<property name="KawaHome" 			value="${KAWAHOME}" />
<property name="WebLogicHome" 		value="${WEBLOGICHOME}" />
<property name="JDKHome" 			value="${JDKHOME}" />
<property name="ArchiveHome" 		value="${ARCHIVEHOME}" />

<!--
	Build target indicates which of the styles of build is desired
	The choices here should be debug, release, coverage.
-->

<property name="BuildTarget" 		value="${BUILDTARGET}" />

<!--
    Lots of other properties like classpaths and such get defined
-->

...
==========================================================

\uber\project-1\build\project.ant simple set up dependencies between
targets and delegated to the jar.ant and ejb-jar.ant files

A jar.ant file included a bunch of the base rules from the the \uber\build
folder and defined some properties, paths and filesets to be used by those
rules.

A typical jar.ant file looked something like:

==========================================================

<?xml version="1.0"?>
<!DOCTYPE project [
    <!ENTITY GlobalConfig SYSTEM "file:../../../../../../build/config.ant">
    <!ENTITY DocLintTarget SYSTEM
"file:../../../../../../build/jar.doclint.target.ant">
    <!ENTITY PrepareTarget SYSTEM
"file:../../../../../../build/jar.prepare.target.ant">
    <!ENTITY CompileTarget SYSTEM
"file:../../../../../../build/jar.compile.target.ant">
    <!ENTITY JTestTarget SYSTEM
"file:../../../../../../build/jar.jtest.target.ant">
    <!ENTITY CollectJavadocTarget SYSTEM
"file:../../../../../../build/jar.collectjavadoc.target.ant">
    <!ENTITY CleanTarget SYSTEM
"file:../../../../../../build/jar.clean.target.ant">
    <!ENTITY AssembleTarget SYSTEM
"file:../../../../../../build/jar.assemble.target.ant">
    <!ENTITY UnitTestClassesTarget SYSTEM
"file:../../../../../../build/jar.unittestclasses.target.ant">
    <!ENTITY UnitTestDeployablesTarget SYSTEM
"file:../../../../../../build/jar.unittestdeployables.target.ant">
    <!ENTITY DeployTarget SYSTEM
"file:../../../../../../build/jar.deploy.target.ant">
]>

<project name="clearance"  default="compile">

	<!--
		Includes the common configuration items
	-->

	 &GlobalConfig;

	<!--
		Given Information about the items to be built.
		May need to be changed upon project creation.

		parentProject: The name of the parent "project" that owns
		the "subproject" that owns this jar.

		parentPackage: The name of the parent package that "owns" this jar.

		packageList: The comma separated list of packages to be
		documented. Used by the javadoc rules
	-->

	<property name="parentProject" value="singular" />
	<property name="parentPackage" value="diamond.payor.credit" />
	<property name="parentPackagePath" value="diamond/payor/credit" />

	<property name="packageList"
value="${parentPackage}.${ant.project.name}.**" />
	<property name="packagePath"
value="${parentPackagePath}/${ant.project.name}/**" />

	<property name="testcasePackage"
value="${parentPackage}.testcases.${ant.project.name}.**" />
	<property name="testcasePackagePath"
value="${parentPackagePath}/testcases/${ant.project.name}/**" />

	<!--
		Given information that rarely needs to be changed , even when the project
is created.

		srcDir: The source directory for the subproject that owns this jar. This
should almost never need to be modified.

		jarName: The name of the jar to be created
	-->

	<property name="srcDir" value="${basedir}/../../../Src" />
	<property name="jarName" value="${parentProject}.${ant.project.name}.jar"
/>

	<!--
		Source Directories

		This is the complete set of source directories that contain any source
		that we may need to compile
	-->

	<path id="jar.src.paths.patternset" >
		<pathelement path="${srcDir}"/>
	</path>

	<!--
		Package Paths

		These are the paths that describe the packages to be included
		They are expressed using '*' so that they will match either classes
		or source files.
	-->

	<!-- This is the package -->

	<fileset dir="${basedir}" >
		<patternset id="jar.package.path.patternset" >
			<include name="${packagePath}"/>
		</patternset>
	</fileset>

	<!-- This is the package path of the classes the EJBs are dependent on. Use
a "." if there are none -->

	<fileset dir="${basedir}" >
		<patternset id="jar.dependent.package.path.patternset" >
			<include name="."/>
		</patternset>
	</fileset>

	<!-- This is the package path of the classes to be unit tested -->

	<fileset dir="${basedir}" >
		<patternset id="jar.testcases.package.path.patternset" >
			<include name="${testcasePackagePath}"/>
		</patternset>
	</fileset>

    <!-- Include the common unittestclasses target -->
    &UnitTestClassesTarget;

    <!-- Include the common unittestdeployables target -->
    &UnitTestDeployablesTarget;

    <!-- Include the common deploy target -->
    &DeployTarget;

	<!-- Include the common assemble target -->
	&AssembleTarget;

	<!-- Include the common clean target -->
	&CleanTarget;

	<!-- Include the common collectjavadoc target -->
	&CollectJavadocTarget;

	<!-- Include the common compiler target	-->
	&CompileTarget;

	<!-- Include the common doclint target 	-->
	&DocLintTarget;

	<!-- Include the common jtest target -->
	&JTestTarget;

	<!-- Include the common prepare target -->
	&PrepareTarget;

</project>


And a typical included jar target such as jar.assemble.ant is:
==============================================================

<!--
	The assemble target

	Assembles all the classes into a single jar
-->

<target name="assemble" depends="compile,prepare">

	<echo>Assembling (${jarName})</echo>

	<jar jarfile="${ProjectAssemblyDir}/${jarName}" >
		<fileset dir="${ProjectClassDir}" >
			<patternset refid="jar.package.path.patternset" />
		</fileset>
		<fileset dir=".." includes="META-INF/**" />
	</jar>

</target>


==============================================================



Evaluation
==========


Pros...
--------

That first system worked and worked fairly well. We could invoke the ANT
files
from within KAWA at the master, project, subproject, and jar levels and all
the
dependencies worked out fairly well.

Cons...
-------

Speed. The fact that we walked the dependency tree at each level (master,
project, etc..) made things dog slow.

In case it is not obvious why consider the walk down through the master and
project and so on for the assemble target (makes Jars).

1) Master.ant (assemble)->(compile)->(prepare)

2) Master.ant (assemble)->(compile)->(prepare)
                             |
                             V
                          project.ant (compile)
                             |
                             V
                          subproject.ant (compile)
                             |
                             V
                          jar.ant (compile)

3) Master.ant (assemble)->(compile)->(prepare)
                 |
                 V
               project.ant (assemble) -> (compile)
                                             |
                                             V
                                          subproject.ant (compile)
                                             |
                                             V
                                          jar.ant (compile)


4) Master.ant (assemble)->(compile)->(prepare)
                 |
                 V
               project.ant (assemble)
                 |
                 V
               subproject.ant (assemble) -> (compile)
                                             |
                                             V
                                          jar.ant (compile)


And so on. Can you say Geometric growth ?

Each project or subproject we added
made it worse. As the project got bigger we started waiting 2 hours just for
the build without testing.

In addition, the Source Control system was very slow and we
were pulling down anew each time to ensure that it was a clean build.

The PERL scripts did not grow well with the system as we changed
configuration
and we had duplicate information between the PERL script config files and
ANT build files. Some interesting snafus resulted.


The solution was evolutionary but I will present it as if we made it in one
fell
swoop.

We now have a simpler directory structure.

The directory structure originally looked like this:

\<uber project>

    \build  - contains buildfiles (master.ant) and configuration files

    \builds
        \<appserverver>
            \<compilationconfig>
                \classes        - root of the compiled Java output
                \javadocs       - root of the Javadoc output
                \deployment     - For holding deployables like
EARs,WARs,JARs

    \<project1>
        \build                  - contains project build file project.ant
        \ears
            \<ear-name-1>
                \WEB-INF        - deployment descriptors

        \wars
            \<war-name-1>
                \WEB-INF        - deployment descriptors

        \development
            \<subproject-1>
                \src            - root of subprojects source tree
                \jars
                    \<jar-name-1>
                        \META-INF

                \ejb-jars
                    \<ejb-jar-name-1>
                        \META-INF

We eliminated the subproject, jar, and ejb-jar ant files and the build
directories that contained them.

The master.ant file calls various project.ant files that are fairly self
contained
build units that make all of the EARs, WARs, JARs and other deployables.

We changed over to using Perforce as our revision control system.

We changed over to using "Cruise Control" as our controller for the periodic
builds.


Evaluation
==========

Pros
----

The system now is significantly faster, with a full build including
pulling down the source tree taking no more than 20 minutes (without
testing).
Cruise Control does most of what we need in terms of automation.
And I modified Cruise Control so that I can add other properties to its
config
files so that I can pass in a complete build configuration to teh ANT
scripts.

Maintainence is now much easier

Cons
----
Developers can no longer do a build at any level they want.
Most have no problem with this because the speed is overall
much better.

It does the job but there are small things within ANT that
required me to hack things in various ways.

Logging of what happens is stil somewhat ungainly and ugly.

We are somewhat at a loss when we want to choose whether or not
a failed test or a failed deployment constitutes a faiilure of the build.
Since we label a source tree based on a successful build I would love to
have (for any target) the ability to specify whether or not a failure of
this
target causes the build to stop.

The fact that JAR files cannot be updated easily causes a world of
workarounds.

The ability to pass in a bundle of configuartion info (a properties file?)
into
the build command line would help

The existing APIs for using ANT as a Java tool were a little unwieldy. I
would
probably have written a Kawa plugin for using ANT if it had been cleaner.

I am sure that there are more but hopefully this wil give everyone a feel
for
the fun stuff I have been doing and save some poor schmoe a few hours.

Jim Jackl-Mochel


--
To unsubscribe, e-mail:   <mailto:ant-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:ant-user-help@jakarta.apache.org>


Mime
View raw message