ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Vogel <pvo...@arsin.com>
Subject RE: Whoa Bessie... Was -- Re: [Proposal] AntFarm
Date Tue, 19 Dec 2000 00:06:26 GMT
Good response Peter!  Some comments below...

> From: Peter Donald [mailto:donaldp@apache.org]
> At 01:02  18/12/00 -0800, Peter Vogel wrote:
> >I wonder, how many of the current ant developers have looked 
> at "other"
> >alternative build tools (by alternative, I mean non-"make" 
> based) like
> >CONS, etc.?
> 
> Not me ;) I may have to go do a search on google when I get 
> some time thou ;)

I'll save you some time: www.dsmit.com/cons/ 

> 
> >Having spent the last 12 years with CM representing at least 
> a part of
> >my responsibilities, and the last 6 with CM being my primary 
> specialty,
> 
> You realized that you may have just volunteered for brain picking ? ;)

What else is new? :-)  Yes, I did realize what I was doing, getting my
brain picked is a regular part of my job responsibilities :-)

> 
> >I am surprised at the lack of "cross fertilization" of ideas 
> and concepts
> >between build tools.  
> 
> lack of knowledge about each other I guess.

So I feared...

> 
> >Ant has some great concepts (xml based build file, 
> >somewhat object-oriented and extensible) but is immature and 
> pretty limited
> >in its ability to manage a LARGE codebase,
> 
> I am not sure if you mean LARGE or COMPLEX ;) I run/help 
> setup quite a few
> LARGEish projects. However they were relatively simple build 
> processes that
> don't deviate too much from traditional model. 
> 
> However ask Diane - there are a lot of things that make it 
> difficult to
> setup complex build environments. There is a little lack of 
> control wrt
> setting all execution values as many aren't fully exposed. 
> (ie Javac and
> extdirs with some compilers)

I meant complexity, absolutely, which frequently (but as you
point out not always) is implied by a large project.

> 
> > especially where parts of the 
> >codebase are shared between projects, 
> 
> Right - until I saw the AntFarm proposal I would have 
> advocated monolithic
> build files now I think I have changed my mind ;)

Yep!

> 
> >it's also pretty Java-focused.  CONS
> >is very powerful with the full expressive power of PERL, built-in and
> 
> you have to remember that Ant has deliberately sacraficed 
> power in some
> places to make it simple and easy to use. We still haven't 
> found the exact
> balance between power/ease/expressability but we try. Many of 
> the things in
> make (and I guess CONS ???) would never make it into ant as 
> it is too complex.

At times, it requires complexity to support a complex project :-)  I think
the key, really, is to have a simple project have a simple build.xml file,
and
a complex project will necessarily require a more complex set of features.  
To sacrifice functionality to KISS is not a long-term strategy for success!
Projects tend to grow, organically, both in terms of size and complexity,
the 
build environment needs to support that growth.

> 
> >extensible
> >dependency scanning, non-recursive build, etc. but is 
> currently fairly C/C++
> >focused, though certainly less so than Ant is Java focused.  
> 
> The beuaty of ant is that it is all in the tasks. You want 
> c/c++ tasks then whip em up ;)

Yes, CONS works in much the same way...

> 
> >Support for detailed dependency checking, including 
> "non-traditional" 
> >dependencies such as which compiler is in use now vs. the last time 
> >a compile was done in the tree.   
> 
> Okay - I think these should be placed in tasks rather than in core.
> Delicate spiderwebs of dependencies that make builds are 
> unacceptable - I
> much prefer simplity of if/unless + tasks to set properties.

No spiderwebs implied!  I was thinking in terms of CONS' (and several
proprietary make-based tools') use of MD5 checksums on the files 
and options involved in a build step to ensure that a change in something
propagates appropriately through the tree.  (For example, if a library 
changes, ensure that anything that links with that lib is re-linked).

> 
> >Automatic dependency update and verification, that is, no 
> "make depend" or
> >equivalent.
> 
> agreed. Thou currently we rely fairly strongly on good 
> compilers this is
> not always a good thing - then there is Conors depends but I 
> am not sure if
> that will scale well to thousands/millions of files 
> (especially when you
> use inner/anonymous classes a lot).

Right.

> 
> >Support for building from anywhere in the tree, updating only what
> >is necessary to build at that level and below.
> 
> Not sure I agree. I think I prefer AntFarms idea of small 
> reusable project
> files that depend on other projects is a better model. 
> Directory sub-builds
> is just make et als poor attempt to do the same. Make assumes that
> sub-directories assume sub-components which is not always 
> case especially
> in some languages (ie java ;]).

I didn't mean to imply a make-based concept here.  It's just that if you
watch a developer work (especially when they aren't working within an IDE,
you will see them change some files, rebuild, test, etc. if they have to CD
to a top level (or switch terminal windows) in order to rebuild, it's an 
interruption to the "flow".   So, if one implements the concept of an upward
search for build.xml you're half-way there, then add the idea that if
they are in the directory foo/bar/baz then they want to rebuild starting
from the target foo.bar.baz (obviously checking dependencies, etc.) until a
particular set of "goal" targets are completed, you have the right idea...


> 
> >Support for multiple projects sharing code within a single tree.
> 
> agreed.
> 
> >Support for embedded information.  Every CM environment is 
> different, and
> >a regular build process has different notification requirements from 
> >location to location, if this information can be maintained in common
> >with the build script(s) themselves, all the better.
> 
> I agree - essentially in Ant2.0 from what I understand the 
> main build file
> will be in the "ant" namespace and anyone can tack anything 
> on in other
> namespaces.

I just wonder about the need to add the complexity of namespaces here...
See my earlier messages about extensible attributes...

> 
> >Support for a "distribution" build that extracts only those 
> derived objects,
> >etc. that are required for installation of the built 
> components on a target
> >deployment system.
> 
> Thats a build pattern - not something that requires direct 
> support in tool.
> Just like most make/ant projects have usage and clean targets 
> we should
> establish conventions for "dist" targets.

Agreed, again as I said earlier the point is to not preclude 
the functionality and to ensure that the *concept* is there
to be implemented as needed by each project.

> 
> >Support for automation by an external entity.  It should be 
> possible to 
> >conduct an "unattended" build on a regular basis.  This 
> *should* imply
> >a command-line interface.
> 
> agreed.
> 
> >Support for different build "configurations" -- the simplest 
> example of this
> >is a debug vs. a release build, but there are others: in one 
> example from my
> >past we had 24 different build configurations 
> (Release/Debug, DVD Region
> >1/2/3/4/5/6, Eval/Paid). All should be buildable/distable 
> within a single
> >tree (so
> >derived objects should be maintained in a separate location 
> from the source,
> >etc.)
> >While less of an issue for Java, target HW/OS differences may also be
> >considered a configuration differentiator, as might compiler 
> or compiler
> >version).
> 
> I think that will be possible by just changing values of properties at
> start of build and running it (assumedly you would point it 
> at different
> destination directories). This will be possible with new 
> proposals who are
> fully interpreted thou it should be most possible with 
> current ant aswell.

The key here is for certain "standard and consistent" property value sets 
to be available as named configurations, including the destination directory
names, etc.  One of the nightmares of CM is the version of a build from
an engineer's machine that "works for me" but the official CM-build version
of the build fails in some way or another.  Sources are identical, what else
changed?  By ensureing that there are known, understood, controlled
configurations
that should be built, you reduce and/or eliminate the engineer's build where
the engineer made one "tweak" to the make options when they built and have
since forgotten (or don't want to admit) that they did a "local tweak".  So,
by encapsulating named configurations of consistent property values, you
define
the set of "standard" builds in a form that is a part of the source tree
(because it is in the build.xml file) so you detect any local tweaking with
a 
simple source compare, and you constrain the set of configurations that need
to be built and tested.

> 
> >Support for integration into relevant IDEs (this could be 
> subsumed by my
> >"automation by an external entity" req. but I felt it worth 
> being explicit)
> >that is, the tool should support being integrated into the 
> developer's
> >workflow,
> >not the developer integrating into the tool's workflow.
> 
> right.
> 
> >Support for automated parsing of build logs -- it should be 
> possible to
> >identify
> >the location of any build errors that occur, where building for that
> >location
> >started and ended, and what errors occurred.  Obviously some 
> support from 
> >compilers and other tools is to be expected here, but help 
> from the build
> >tool
> >is useful, for example, most tools exit with a non-zero 
> return code if
> >something
> >went wrong -- the build tool should provide output 
> indicating what it tried
> >to
> >do (flags, etc.) and the RC it got if it gets a non-zero return code.
> 
> agreed. Most tools follow emacs conventions for error detections so we
> should formalize on that.
> 
> >Support for *any* tool to be called from the build. 
> 
> anything that can be programmed via a task(let) - sure.

I guess this is where there is some philosophical difference: one should
not have to program an extension to the build tool for every command one
might need.  Otherwise, you run into a problem of extensions that do
*almost*
the same thing, etc.  CM nightmare!

> 
> 
>  For example, I like to 
> >include a "build number" in all derived objects, that build number is
> >usually
> >embedded in a header file included as needed by other source 
> files, but we
> >do
> >not like to check in 100's of revisions to a header file 
> just to update a
> >build
> >number, so usually the header file contains a "placeholder" 
> for the build
> >number
> >and that number is updated automatically by the build process in a
> >raw->cooked
> >pre-preprocessing step usually done by a tool like perl/sed/whatever.
> 
> It is already possible with a little bit of javascript + 
> replace task ;)

Yes.  Again, the requirement is there to ensure that it is supported
and not precluded.

> 
> >Support for both centralized and de-centralized (but 
> non-recursive) build 
> >description files.  In a large project, having a single 
> build description 
> >file implies a huge number of revisions to that one file, 
> because a change
> >anywhere in the tree might require a change to the build 
> description, it is
> >usually better to distribute the information about how to 
> build a region of
> >a tree into that region of the tree.  That way information 
> about a subtree
> >is local to that subtree.  This DOES NOT imply a 
> recursive-style build (i.e.
> >the traditional "cd subdir; make" approach) only the localization of
> >information
> >implied by that.  
> 
> Have a look at AntFarm as I think you will be pleasantly surprised ;)

I took a quick look, I agree, this is the right idea!  CONS does this
extremely well also.

> 
> >At the same time, there is a huge need for some
> >information
> >(i.e supported build configurations, how to perform various 
> common tasks
> >(compile,
> >etc), etc.) to be centralized for ease of maintenance of the 
> whole tree.
> >(I've
> >done this many times taking advantage of GNUmake's "include" 
> directive to
> >include
> >centralized variables and targets, while individual 
> makefiles just provide
> >their
> >specifics, like what files to compile and what target to 
> assemble) CONS does
> >this
> >real well with a top-level file that is always grabbed first 
> (no matter
> >where
> >one invokes cons from within the tree) and sub-files 
> incorporated as needed.
> 
> What you are talking about here (templated build files) has 
> been discussed.
> While it is possible it is not something we will directly 
> support (because
> of added complexity). What I want to do is do templates via 
> template file +
> xslt --> build.xml. Not sure on viability yet.

Again, don't fear complexity!  Just make sure that one needs to get
complex only when one needs the functionality!  For example, a simple
"hello world" C program for windows is incredibly complex if you create
your output window, load font, etc.  Microsoft really blew it there (in 
the early days of Windows) by forcing the developer to delve deep into
the complexity of windows for even simple things.  Contrast to the Mac's
MPW environment, which allowed you to write the traditional hello world
program and the built program created its own window for input/output, etc.
and as the developer needed the capabilities of a windowed program they
could accept the complexity of programming in an event-driven environment.
The same concept applies to build tools, the POWER should be there, when 
needed, but should not get in the way of writing the simple build file
to support your basic Java "hello world" program.

> 
> >Support for building and exporting unit and integration test 
> information
> >(scripts,
> >data, test benches, harnesses, etc.) 
> 
> just tasks. Something we already have now aswell ;)
> 
> >Support for tools like code coverage analysis tools that 
> have a "precompile"
> >component which generate data files that need to be 
> harvested for the 
> >analysis phase later, etc.
> 
> some of that support just requires tasks and no direct 
> support from ants
> core. However there is need for small globs of "temporary" 
> information. In
> this case it would be useful to assign result data to 
> ant-properties and
> retrieve it in a later task. I have made that possible in 
> myrmidon for this
> reason.

You definitely get my point!  

> 
> >Again, this is based on 12 years of doing CM for projects 
> ranging from a 10K
> >lines of code to 2M lines of code with varying degrees of 
> code sharing, and
> >the need to support 100s of developers, etc.  I've done my 
> best to avoid 
> >describing any "hows" here, just "whats" but I'd be happy to 
> discuss hows
> >with
> >anyone interested, based on my experience implementing many 
> of the above
> >requirements with a variety of tools including Make (GNU), 
> Perl (alone and
> >in conjuction with other tools), CONS (perl-based framework), and, to
> >a lesser extent, ANT.
> 
> The only thing you have to realize is that Ant is not meant to be a
> cure-all it is waaaay to early in development to try to do that yet.

I have to ask myself: Why build a build tool if it cannot meet the 
needs of real projects?    Even if you don't implement features immediately,
the tool must be *designed* to grow toward a powerful build tool that meets
the needs of complex projects.

> Simplicty and usability are key requirements unlike other 
> build tools that often require a make wizard/CM to understand.

Again, to the philosophical:  I can write a simple make file for a simple
project, and it takes no wizard to understand what I did.  But if I have
a complex project that I need to ensure is built in a repeatable and
consistent fashion then I probably need a complex build tool to meet the
needs of the project, and the project is probably complex enough to 
imply the needs of a CM specialist who understands the build tools!

The problem comes because projects *always* start simple, but grow as
time and requirements grow (look at ANT :-) and if the build environment
cannot grow with the project, then you force the team to switch to a wholly
different build environment and you *guarantee* that no-one but the CM
who did the conversion work understands the build environment!  If you 
support gradual discovery of power, then you support an environment where,
perhaps, not everyone understands everything about the build environment,
but
where everyone understands enough to make the changes they need to make and
knows to ask someone who knows how to do the things they don't know how to
do.

-Peter

Mime
View raw message