ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse Glick <Jesse.Gl...@netbeans.com>
Subject Re: Incremental Compilation and ant
Date Fri, 21 Jul 2000 13:50:56 GMT
Warning: this is a long message...

Conor MacNeill wrote:
> 2. Add a separate dependency task which generates a dependency file. Javac
> would use that file to determine which files would need to be recompiled. I
> proposed a patch for this approach. It decouples the dependency analysis
> from the javac task

I'd like to add a suggestion for a variant of this. Basically it would
be one task, not two, which runs an incremental compilation and also
computes dependencies. Its advantages are that it should work with any
Java compiler, no matter how stupid its dependency analysis; it would be
transparent for the user to invoke (just like a normal compilation task,
give the source directory, non-source classpath, and a compiler version
to use); should (re-)compile exactly those files it needs to; should
behave sanely even if compilation of the same sources is done in the
interim without the dependency-tracking task (e.g. bare command-line
compiler). I wrote a Perl script a while back which mostly implements
it, though that script was kind of a mess and it would be much cleaner
as an Ant task.

Setup: somewhere there should be storage of dependencies. In my script,
.java and .class files lived in the same directory (no -d option), and I
added .dep files for each source as well in the same directory. Each
.dep file was a text file with a list of dependencies, one per line,
given as class names (outer classes only, no inner classes listed); it
represents the dependencies in the source tree of the given class, not
to include classes present in JARs and so on in the (non-source)
classpath, nor including the class itself. Timestamp of the .dep file is
significant. For efficiency, it would be possible to store a single deps
database in some format, listing timestamp, dependent class name, and
all source dependencies of that class. Or .dep files could be placed in
a special separated directory.

Algorithm:

1. Prepare list of classes to compile. Scan the source tree (given
includes/excludes, etc.) for source files. A source file is out of date
if any of the following are true:

1a. It has no .class file.

1b. Its .class file is older than the source file.

1c. It has no .dep file.

1d. Its .dep file is older than the source file.

1e. One or more of the source files listed in its .dep file is missing.

1f. One or more of the source files listed in its .dep file is newer
than the considered source file.

2. Run the selected Java compiler on the source files thus gleaned. It
should not matter how that compiler does dependency analysis, or even if
it does any; all required files ought to have been explicitly listed
anyway.

3. Again using the same list of source files, find the freshly-compiled
.class files and parse them. A full bytecode library should not be
necessary, only ability to parse the constant pool. All classes
referenced at compile time should be listed as class constants in this
pool. (My original script simply grepped the binary .class file for
likely references using a conservative approximation, but doing it right
should not be that hard.) Remember to search all inner-class files.
(JDK-1.0-style package-private outer classes might be a problem here--it
may be necessary to check the "source" attribute in the class file to
handle these.)

4. Create a fresh .dep file listing all those classes found in the
constant pool which are in fact present in the source tree (vs. library
or JRE classes). Issue a warning if a dependent class is present in the
source tree in .class but not in .java form (and still add a dependency
for it).

Now what happens is this: the first time the project is built, every
class will be given to the compiler since there are no .dep files.
Subsequently the user may modify some source files. On the next build,
certainly these files will be recompiled; any classes directly depending
on these modified classes should also be compiled, according to
information from the .dep files. (Note that unlike C/C++, dependency is
not transitive in Java, i.e. only direct dependencies need be
considered.) So a developer can always ask to compile the whole source
tree and get only recompilations which are necessary, or might be
necessary (in case an API might have been broken). Removals of source
files from the tree should also trigger recompilation of dependent files
(presumably displaying the correct compilation errors). Additions of
source files, or making modifications which change the list of
dependencies for a class, should also be treated correctly. It is safe
to compile a few files by hand, since the next proper build will find
that the .dep files are out of date and rebuild them. It should also be
safe to remove .dep files if you want to.

Note that I think you need to not just generate a .dep file but actually
recompile after just 1c. or 1d. holds, even though it seems you would
not, in order to ensure that removals of source files trigger
recompilations reliably.

It is necessary to make sure that compilation errors do not leave behind
an up-to-date .class file (which hopefully they do not).

There is only one exception I can think of to Java dependencies not
being transitive: a static final primitive/String constant defined in A,
and referenced in B as the value of another constant, and referenced in
C somehow, could make C's compiled form depend directly on A's source,
when using an optimizing compiler. I think this is a pretty rare case
(especially since runtime use of a JIT or HotSpot makes optimizations
like that close to useless). Maybe similar for final method inlining?
Someone should check the JLS on this but I think such dependencies are
officially pretty restricted, and it would be just as well to disable
optimization flags in this kind of situation.

I believe the above algorithm is correct, but as I said it is some time
since I really used it, so there might be some subtle gotchas. Comments
welcome...

Cheers,
-Jesse

-- 
Jesse Glick   <mailto:Jesse.Glick@netbeans.com>
NetBeans, Open APIs  <http://www.netbeans.org/>
tel (+4202) 3300-9161 Sun Micro x49161 Praha CR

Mime
View raw message