ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Donald <>
Subject Re: Smarter Javac
Date Mon, 19 Nov 2001 10:56:09 GMT
On Sat, 17 Nov 2001 19:18, Magesh Umasankar wrote:
> I noticed that this has been discussed for a short while
> a long time back, but nothing solidified though.  So,
> here goes:


> Javac, as we are all aware, determines the file set that
> is to be compiled based solely on comparing the
> timestamps of the source and destinaiton files.  However,
> this has got its limitations in numerous ways - one is
> dependency checking, which has been solved by the
> Depend task.  But still the glaring problem is this:
> Say I compile a bunch of files using debug mode and
> in my next iteration, I build it in non-debug mode,
> without having changed any of my source files. Javac is
> not going to catch this and would ignore processing any
> files (Yes, I am coming from the CONS world ;-) )  This
> problem is going to exist when I switch compilers, or
> when I enable/diable warnings, etc.


> CONS has a nice way of solving this - it caches the
> command line being used and checks against the cached one
> during future compiles to see if the current command line
> matches the earlier one.

I suspect CONS has a slightly easier time in some ways. However I have only 
heard of CONS second hand .. well I did read the docs for it once but 
promptly forgot everything ;)

CONS can do it at the command line level as each command line execution is 
compilation for one "source" file. Ant aggregates the command line and thus 
has a slightly harder time.

> In the context of Ant, here is what I propose:
> A cache file would be maintained that holds key-value
> pairs.  A key-value pair is the task name and the MD5
> checksum of the Javac command line.  When a Javac task
> named foo is invoked, the MD5 checksum of its command
> line is computed and checked against the cache, if
> available, by looking up the key foo.  If the cache isn't
> there, one is created.  If the checksums differ, all
> source files would be compiled again and the new
> checksum would be cached - otherwise, Javac would do
> what it does now.
> While this may slow the Javac process launch just a
> wee little bit, it would be of great use to the user.
> What do you think?  The crude proof-of-concept that I
> built didn't change a lot of code at all...

The problem is that we need to cache the information on a per file basis. 
This can a significant overhead to the compilation step and the exact 
strategy for where (and how) to store this information is really dependent on 
the type of project you are working on.

For small projects it is almost always easier and faster just to do a rebuild 
all every time (assuming of course that you use jikes).

I guess we need to identify the types of "dependency" information. The two 
that we identified last time where 
1. structural
2. coloring

(1) includes things that the file directly depends upon. ie If the .java file 
refers to another class then it has a dependency on the equivelent .java 
file. With c/c++ it would be header dependency data. And sometimes there is 
different types of dependencies ... ie Class A may depend on the public 
interface of class X, protected interface of class Y and package-access 
interface of class Z. There are tools out there that build this information 
for us.

(2) Coloring includes those things you talked about (ie compiler, debug 
status optimizations turned on, etc). It would be up to the javac task to 
identify the "relevent" parameters and chuck them in cache aswell.

Now how to store this information? There is two extremes. One extreme saids 
use a single cache for whole project while the other extreme saids have a 
single dependency cache file per source file. There are also inbetween styles 
(ie cache per package or perhaps per hierarchy). 

However we need to leave it up to the build writer how to store this cache 
information somehow. Some systems have really really really slow file system 
access (ie the dos boxes) so you would want to have caches that contained 
aggregated dependency information. However this will not scale when you have 
large number of files that you may not compile in a single sweep. (ie if you 
use 2 passes of javac for some reason).

It would be nice to have a utility object that could do all this for us. So 
in javacs case you could do something like

DependencyScanner ds = ...;

if( cachePerFileStrategy ) ds.setCacheStrategy( DependencyScanner.PER_FILE );
else ds.setCacheStrategy( DependencyScanner.PER_DIR ); 

ds.addColoring( "debug", "true" );
ds.addColoring( "compiler", "jikes" );
ds.addColoring( "optimize", "false" );
ds.addFileSet( sources1 );
ds.addFileSet( sources2 );

File[] filesToCompile = ds.scan();

doCompile( filesToCompile );

Unfortunately that only works if all the files were compiled successfully. If 
not you have to figure out which ones were updated and which failed and 
update dependency scanner/cache appropriately.

However this level of sophistication may have to wait till ant2 because it is 
a lot of work.



| "Nearly all men can stand adversity, but if you want |
| to test a man's character, give him power."          |
|       -Abraham Lincoln                               |

To unsubscribe, e-mail:   <>
For additional commands, e-mail: <>

View raw message