openoffice-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kay Schenk <kay.sch...@gmail.com>
Subject Re: Bottom up build
Date Thu, 30 Jan 2014 21:55:05 GMT
On Wed, Jan 29, 2014 at 1:54 AM, jan i <jani@apache.org> wrote:

> On 29 January 2014 10:18, Andre Fischer <awf.aoo@gmail.com> wrote:
>
> > I would like to report some observations that I made when thinking about
> > how to make building OpenOffice with one global makefile feasible.  It
> will
> > probably the last of build related mails in the near future.
> >
> > Traditional make uses a top-down approach.  It starts with a target,
> 'all'
> > by default, and looks at its dependencies.  When one of those has to be
> > made or is newer then the target then the target also has to be made.
>  This
> > is done recursively and depth-first.  Every file on which 'all' has a
> > direct or indirect dependency has to be checked.  If we would build
> > OpenOffice with one makefile (included makefiles don't count) then that
> are
> > a lot of files to check.  There are about 9800 cxx files, 3500 c files,
> > 12000 hxx files, and lot of external headers.  Checking the modification
> > time of so many files is one of the reasons for the delay in , say, sw/
> > between starting make and its first action.
> >
> > As I don't have all global dependencies in a format that would allow
> > experimation, I tried how long it would take to get the mtime (time of
> last
> > modification) for all files, source, generated, compiled, linked, about
> > 120000.  I wrote a small Java program for that.  With a warm cache that
> > takes about 23s.  When run in 4 threads this reduced to less than 8s.
> >  Could be worse.
> >
> > But it also could be better because in general there are only a few files
> > modified, usually files that you modified yourself in an editor.  There
> is
> > another approach, introduced, as far as I know, by the tup [1] build
> tool,
> > that is bottom up.  If you had something similar to the oracle of
> > complexity theory, that gave you the list of modified files since the
> last
> > build, you could find the depending files much faster.  Faster for two
> > reasons. Firstly, there is only one path in the dependency tree up
> towards
> > the root (while there are many down from the root).  Only targets on this
> > path are affected by the modified file. Secondly, the dependency analysis
> > is comparatively cheap.  The expensive part is to determine the file
> > modification times.  If they where miraculously given then even the
> > top-down approach would not take noticably longer.
> >
> > So I made another experiment to see if such an oracle can be created.
> >  Java 7 has the java.nio.file.WatchService that lets you monitor file
> > modfifications in one directory.  I registered it to all directories in
> our
> > source tree (some 16000 directories).  With the WatchService in place
> every
> > file modification can be recorded and stored for later.  On the next
> build
> > you only have to check the set of modified files, not all files.
> >  Registering the directory watchers takes a couple of seconds but after
> > that it does not cause any noticeable CPU activity. Any file
> modifications
> > are reported almost at once.  I do not have the framework in place to
> start
> > a build with this information but I would expect it to be as fast as
> > compiling the modified files and linking takes.
> >
> > The tup website references a paper [2] in which the established top-down
> > approaches are called alpha alogithms and the new bottom-up approach is
> > called beta algorithm. Tup has implemented a file modification watcher
> (in
> > C or C++) only for Linux.  On Windows it just scans all files (for which
> it
> > needs a little more time than my Java program, maybe it does not use more
> > than one thread).
> >
> >
> > This is something that we should keep in mind for when we ever should get
> > a build solution with global dependencies and this build tool would turn
> > out to be too slow.
> >
> >
> > If can find the source code of my Java experiments at [3]. If nothing
> else
> > you can see an application of the ForkJoinPool that allowed my to write
> the
> > parallel file system scan in just a few lines.  There is also an
> > alternative implementation that uses the ExecutorService (with a fixed
> > thread pool) which needs a few more lines of code.  And there is of
> course
> > the use of the WatchService.
> >
>
>
It's really interesting to read these observations and test cases... we
have a large and complicated source tree and just seeing what can be
observed about it is fascinating to me.

Thanks for writing down your observations which I find highly interesting.
> I hope your stop on writing about build does not include giving your
> opinion on my ideas in the future as well.
>
> For the record the capstone project, and my little hobby project "Build
> R.I.P." follow a third idea:
>
> We have a clear seperation of  module build and central (total) build.


+1 I would certainly go for this.

 A while back someone asked about ye olde "ld" approach -- all modules
compiled/built and then linked later down the road. If we could somehow do
something to get back to that idea in a more friendly modern way, it would
certainly make working on specific areas more feasible.




> The
> module makefile knows how to build the module, and the central makefile
> knows the relation between modules.
>
> The makefile in each module touched a file, and the central makefile only
> controls that file.
>
> But youir idea of watching for changes is very interesting.
>
> rgds
> jan I.
>
>
> > Andre
> >
> >
> > [1] http://gittup.org/tup/
> > [2] http://gittup.org/tup/build_system_rules_and_algorithms.pdf
> > [3] http://people.apache.org/~af/test.zip
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> > For additional commands, e-mail: dev-help@openoffice.apache.org
> >
> >
>



-- 
-------------------------------------------------------------------------------------------------
MzK

"Cats do not have to be shown how to have a good time,
 for they are unfailing ingenious in that respect."
                                       -- James Mason

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message