openoffice-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damjan Jovanovic <dam...@apache.org>
Subject Thoughts on a new build system for AOO
Date Fri, 05 Feb 2016 18:32:35 GMT
Hi

With the recent buildbot saga and Patricia's Windows building nightmare, a
number of serious issues were highlighted which got me thinking about how
our build system could be improved.

The problems with building AOO were researched and documented years ago (
https://wiki.openoffice.org/wiki/Build_Environment_Effort) and a very good
analysis was done in https://wiki.openoffice.org/wiki/Build_System_Analysis
which is worth a read.

Their solution was gbuild: a build system using GNU make's new "eval"
feature to rewrite the makefile from within, which would generate one giant
dependency graph of every file in AOO and produce reliable, stable,
deterministic, and maximally parallelized builds, with minimal rebuilding
on changes. The problem is, the full benefits of gbuild can only be seen
once every module is converted to gbuild, and with only about 10% having
been converted so far, it doesn't even try to build a complete project-wide
dependency graph, but rather gets invoked per-module just like dmake.
Converting dmake modules to gbuild is very difficult: so far I only managed
to convert main/formula, but main/animations was mysteriously segfaulting
AOO...

Besides, LO has completed converting every module to gbuild, guinea pigging
gbuild perfectly for us ;-). How has it gone for them? One of our
committers, jani, evaluated LO's gbuild in April 2013 in a thread here
called "Make x Dmake x Build", where he said how it's "not particulary fast
either", and called mkdir 431 times on a directory that already existed.
And worst of all, gbuild uses its own language with horrible syntax (eg.
LISP-like syntax, spaces around commas not allowed, blank lines sometimes
mandatory and at other times forbidden), is almost completely undocumented,
and very difficult to debug - a maintenance nightmare.

So what other alternatives are there?

There was a comment in
https://wiki.openoffice.org/wiki/Build_Environment_Effort/New_Build_System_Requirements
that got my attention:

"One might consider using Python as a tool for _all_ non-standard build
tasks. Python is easy to distribute and install on all platforms. However,
it is also a very fat dependency and thus it is questionable, if it would
be worth the effort just to get rid of cygwin (or another Unix environment
on windows)."

We already build and use Python, and there is a good build system written
in Python: SCons (http://www.scons.org). It's modern yet well established,
having started in the year 2000, and is used by a number of open source and
commercial projects. It's MIT licensed so we can include it as a
dependency. The build scripts are simple Python scripts and can use the
Python language and libraries, though you don't need to know (much) Python
to use it. It's well documented. It's debuggable and has options to print
dependencies and explain actions taken. It supports all the operating
systems and architectures we support (even unofficially), including Cygwin,
*BSD, Solaris, OS/2. It's "local" package is designed for easy embedding
into a pre-existing project so it runs straight after extraction with no
need to install it system-wide, and is only 445 kB.

It has many features we need or would benefit from:
* It's able to parse included headers from C/C++ files and use them as
dependencies.
* It can generate Visual Studio project files.
* Supports many languages including C, C++, D, Java, Fortran, Yacc, Lex.
* Builds zip and tar archives.
* Just like gbuild, it has a global dependency view and can fully
parallelize building files across module boundaries while maintaining
stability.
* Using Python APIs instead of *nix CLI tools might eventually result in us
being able to ditch Cygwin entirely, not to mention better performance due
to less subprocess spawning.
* Unlike classical make tools which only use mtimes, it determines what to
rebuild using several configurable options, including MD5 checksums of file
contents, meaning changes which don't result in different binaries (eg.
formatting or comment changes) will only recompile changed files and stop
there, because it knows the object files are the same as before, so no
linking or downstream rebuilding needs to be performed.

In other words SCons does most of what gbuild is supposed to be good for,
and more. I see it outperforming LO's pure gbuild, while being more
readable, maintainable, debuggable, and a sight for sore eyes after the
triple whammy of dmake, gbuild and build.pl we've endured for so long.

If people are happy, I'd suggest a migration plan where first scons is
added as a dependency, then the new version of serf is built with it (see
i126312), then gbuild modules are ported to scons until gbuild is
eliminated, then dmake modules are ported to scons until dmake is
eliminated, and finally scons replaces build.pl.

Thoughts?

Damjan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message