Return-Path: Delivered-To: apmail-jakarta-ant-user-archive@apache.org Received: (qmail 28964 invoked from network); 13 Mar 2002 20:30:59 -0000 Received: from unknown (HELO nagoya.betaversion.org) (192.18.49.131) by daedalus.apache.org with SMTP; 13 Mar 2002 20:30:59 -0000 Received: (qmail 24343 invoked by uid 97); 13 Mar 2002 20:30:44 -0000 Delivered-To: qmlist-jakarta-archive-ant-user@jakarta.apache.org Received: (qmail 24302 invoked by uid 97); 13 Mar 2002 20:30:43 -0000 Mailing-List: contact ant-user-help@jakarta.apache.org; run by ezmlm Precedence: bulk List-Unsubscribe: List-Subscribe: List-Help: List-Post: List-Id: "Ant Users List" Reply-To: "Ant Users List" Delivered-To: mailing list ant-user@jakarta.apache.org Received: (qmail 24291 invoked from network); 13 Mar 2002 20:30:43 -0000 From: "Jim Jackl-Mochel" To: "Ant-User" Subject: My experiences with doing a large project with ANT Date: Wed, 13 Mar 2002 15:26:48 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Introduction ------------ Some people on this list have asked me to post details of how I got a hierarchical ant based build system working for a project I am involved in. This is a simple attempt to get most of my experiences down in a way that might be useful to the ANT community at large. For brevity, I am intentionally leaving out most of the numerous false starts and blind paths I went down. I am detailing out the general rationales and notable blind alleys whenever I see a chance to save someone else a great deal of time. The project is fairly large: 60+ engineers distributed among three locations in the US. It is a J2EE project with multiple application servers and webservers. There were several projects under a single large umbrella project with some code dependencies among them. At the point that I started the project I had a solid experience with automated build and testing in C/C++ and solid Java programming so I didn't figure this would be particularly tough. If anything, I thought that the flexibility and introspective nature of java and java tools would make it even easier. Requirements ------------ We had several requirements that made ANT seem like a godsend 0. We needed to support multiple version of the app server and several different compilation configurations (debug, release, and performance ) 1. We wanted to do automated "overnight" builds that: 1.1 pulled down and labeled the source tree 1.2 compilation 1.3 Unit tested the standalone Java classes 1.4 Packaged the J2EE components for deployment 1.5 Deployed and tested the deployable components 1.6 Run suppoort tools like JTest, Doclint, and javadoc. 1.7 Archived the resulting deliverables 1.8 Mailed the results as necessary 3. We needed to support multiple projects under the same umbrella uber project. Basic History ------------- We defined a directory structure and basic build structure and went to town. We were using Kawa as the IDE of choice. We were running a PERL script in a NT "at" command at various intervals to invoke ANT and send out the mail as necessary. Over time we ran into problems with build time scaling, revision control tools (speed and features), maintainability of the build files and getting configuration information into the ANT build system in the first case. The directory structure originally looked like this: \ \build - contains buildfiles (master.ant) and configuration files \builds \ \ \classes - root of the compiled Java output \javadocs - root of the Javadoc output \deployment - For holding deployables like EARs,WARs,JARs \ \build - contains project build file project.ant \ears \ \WEB-INF - deployment descriptors \wars \ \WEB-INF - deployment descriptors \development \ \build - contains subproject buildfile subproject.ant \src - root of subprojects source tree \jars \ \build - contains jar.ant \META-INF \ejb-jars \ \build - contains ejb-jar.ant \META-INF The directory structure involved a large number of design needs that I will try and summarize. We avoided a source directory off of the root \uber folder because we had many projects and sub projects where the code was worked on offsite, in isolation and we wanted to avoid needing to pull the whole tree down for someone working on a subset of the files. We used the ..\build folders in an attempt to enable automatic generation of ANT files and inclusions into ANT files for hiearchical builds. We named the diferent levels of ant files differently for ease in making sweeping changes to the different files. The WEB- INF and META-INF folders were allocated on a per ejb-jar or jar basis to eliminate name collision or copying schemes during the build. A total build went something like this: ANT was invoked against \uber\build\master.ant. It used ENTITY references to include basic templated targets and actions and other configuration information. master.ant then invoked the project.ant files in order for the different targets (such as compile, testclasses, assemble, package, deploy, testdeployables, etc.. The project.ant files then invoked the subproject.ant files in the same way and they invoked the jar.ant and ejb-jar.ant files. Each of the subfiles themselves included the various rules, targets, and other information from the \uber\build directory. The \uber\build\master.ant file looked like (in summary): ========================================================== ]> &Config; &PrepareTarget; ========================================================== The \uber\build\config.ant file looked like (in summary): ========================================================== ... ========================================================== \uber\project-1\build\project.ant simple set up dependencies between targets and delegated to the jar.ant and ejb-jar.ant files A jar.ant file included a bunch of the base rules from the the \uber\build folder and defined some properties, paths and filesets to be used by those rules. A typical jar.ant file looked something like: ========================================================== ]> &GlobalConfig; &UnitTestClassesTarget; &UnitTestDeployablesTarget; &DeployTarget; &AssembleTarget; &CleanTarget; &CollectJavadocTarget; &CompileTarget; &DocLintTarget; &JTestTarget; &PrepareTarget; And a typical included jar target such as jar.assemble.ant is: ============================================================== Assembling (${jarName}) ============================================================== Evaluation ========== Pros... -------- That first system worked and worked fairly well. We could invoke the ANT files from within KAWA at the master, project, subproject, and jar levels and all the dependencies worked out fairly well. Cons... ------- Speed. The fact that we walked the dependency tree at each level (master, project, etc..) made things dog slow. In case it is not obvious why consider the walk down through the master and project and so on for the assemble target (makes Jars). 1) Master.ant (assemble)->(compile)->(prepare) 2) Master.ant (assemble)->(compile)->(prepare) | V project.ant (compile) | V subproject.ant (compile) | V jar.ant (compile) 3) Master.ant (assemble)->(compile)->(prepare) | V project.ant (assemble) -> (compile) | V subproject.ant (compile) | V jar.ant (compile) 4) Master.ant (assemble)->(compile)->(prepare) | V project.ant (assemble) | V subproject.ant (assemble) -> (compile) | V jar.ant (compile) And so on. Can you say Geometric growth ? Each project or subproject we added made it worse. As the project got bigger we started waiting 2 hours just for the build without testing. In addition, the Source Control system was very slow and we were pulling down anew each time to ensure that it was a clean build. The PERL scripts did not grow well with the system as we changed configuration and we had duplicate information between the PERL script config files and ANT build files. Some interesting snafus resulted. The solution was evolutionary but I will present it as if we made it in one fell swoop. We now have a simpler directory structure. The directory structure originally looked like this: \ \build - contains buildfiles (master.ant) and configuration files \builds \ \ \classes - root of the compiled Java output \javadocs - root of the Javadoc output \deployment - For holding deployables like EARs,WARs,JARs \ \build - contains project build file project.ant \ears \ \WEB-INF - deployment descriptors \wars \ \WEB-INF - deployment descriptors \development \ \src - root of subprojects source tree \jars \ \META-INF \ejb-jars \ \META-INF We eliminated the subproject, jar, and ejb-jar ant files and the build directories that contained them. The master.ant file calls various project.ant files that are fairly self contained build units that make all of the EARs, WARs, JARs and other deployables. We changed over to using Perforce as our revision control system. We changed over to using "Cruise Control" as our controller for the periodic builds. Evaluation ========== Pros ---- The system now is significantly faster, with a full build including pulling down the source tree taking no more than 20 minutes (without testing). Cruise Control does most of what we need in terms of automation. And I modified Cruise Control so that I can add other properties to its config files so that I can pass in a complete build configuration to teh ANT scripts. Maintainence is now much easier Cons ---- Developers can no longer do a build at any level they want. Most have no problem with this because the speed is overall much better. It does the job but there are small things within ANT that required me to hack things in various ways. Logging of what happens is stil somewhat ungainly and ugly. We are somewhat at a loss when we want to choose whether or not a failed test or a failed deployment constitutes a faiilure of the build. Since we label a source tree based on a successful build I would love to have (for any target) the ability to specify whether or not a failure of this target causes the build to stop. The fact that JAR files cannot be updated easily causes a world of workarounds. The ability to pass in a bundle of configuartion info (a properties file?) into the build command line would help The existing APIs for using ANT as a Java tool were a little unwieldy. I would probably have written a Kawa plugin for using ANT if it had been cleaner. I am sure that there are more but hopefully this wil give everyone a feel for the fun stuff I have been doing and save some poor schmoe a few hours. Jim Jackl-Mochel -- To unsubscribe, e-mail: For additional commands, e-mail: