openoffice-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kay Schenk <kay.sch...@gmail.com>
Subject Re: Thoughts on a new build system for AOO
Date Sat, 06 Feb 2016 17:38:11 GMT


On 02/05/2016 10:32 AM, Damjan Jovanovic wrote:
> Hi
> 
> With the recent buildbot saga and Patricia's Windows building nightmare, a
> number of serious issues were highlighted which got me thinking about how
> our build system could be improved.
> 
> The problems with building AOO were researched and documented years ago (
> https://wiki.openoffice.org/wiki/Build_Environment_Effort) and a very good
> analysis was done in https://wiki.openoffice.org/wiki/Build_System_Analysis
> which is worth a read.
> 
> Their solution was gbuild: a build system using GNU make's new "eval"
> feature to rewrite the makefile from within, which would generate one giant
> dependency graph of every file in AOO and produce reliable, stable,
> deterministic, and maximally parallelized builds, with minimal rebuilding
> on changes. The problem is, the full benefits of gbuild can only be seen
> once every module is converted to gbuild, and with only about 10% having
> been converted so far, it doesn't even try to build a complete project-wide
> dependency graph, but rather gets invoked per-module just like dmake.
> Converting dmake modules to gbuild is very difficult: so far I only managed
> to convert main/formula, but main/animations was mysteriously segfaulting
> AOO...
> 
> Besides, LO has completed converting every module to gbuild, guinea pigging
> gbuild perfectly for us ;-). How has it gone for them? One of our
> committers, jani, evaluated LO's gbuild in April 2013 in a thread here
> called "Make x Dmake x Build", where he said how it's "not particulary fast
> either", and called mkdir 431 times on a directory that already existed.
> And worst of all, gbuild uses its own language with horrible syntax (eg.
> LISP-like syntax, spaces around commas not allowed, blank lines sometimes
> mandatory and at other times forbidden), is almost completely undocumented,
> and very difficult to debug - a maintenance nightmare.
> 
> So what other alternatives are there?
> 
> There was a comment in
> https://wiki.openoffice.org/wiki/Build_Environment_Effort/New_Build_System_Requirements
> that got my attention:
> 
> "One might consider using Python as a tool for _all_ non-standard build
> tasks. Python is easy to distribute and install on all platforms. However,
> it is also a very fat dependency and thus it is questionable, if it would
> be worth the effort just to get rid of cygwin (or another Unix environment
> on windows)."
> 
> We already build and use Python, and there is a good build system written
> in Python: SCons (http://www.scons.org). It's modern yet well established,
> having started in the year 2000, and is used by a number of open source and
> commercial projects. It's MIT licensed so we can include it as a
> dependency. The build scripts are simple Python scripts and can use the
> Python language and libraries, though you don't need to know (much) Python
> to use it. It's well documented. It's debuggable and has options to print
> dependencies and explain actions taken. It supports all the operating
> systems and architectures we support (even unofficially), including Cygwin,
> *BSD, Solaris, OS/2. It's "local" package is designed for easy embedding
> into a pre-existing project so it runs straight after extraction with no
> need to install it system-wide, and is only 445 kB.
> 
> It has many features we need or would benefit from:
> * It's able to parse included headers from C/C++ files and use them as
> dependencies.
> * It can generate Visual Studio project files.
> * Supports many languages including C, C++, D, Java, Fortran, Yacc, Lex.
> * Builds zip and tar archives.
> * Just like gbuild, it has a global dependency view and can fully
> parallelize building files across module boundaries while maintaining
> stability.
> * Using Python APIs instead of *nix CLI tools might eventually result in us
> being able to ditch Cygwin entirely, not to mention better performance due
> to less subprocess spawning.
> * Unlike classical make tools which only use mtimes, it determines what to
> rebuild using several configurable options, including MD5 checksums of file
> contents, meaning changes which don't result in different binaries (eg.
> formatting or comment changes) will only recompile changed files and stop
> there, because it knows the object files are the same as before, so no
> linking or downstream rebuilding needs to be performed.
> 
> In other words SCons does most of what gbuild is supposed to be good for,
> and more. I see it outperforming LO's pure gbuild, while being more
> readable, maintainable, debuggable, and a sight for sore eyes after the
> triple whammy of dmake, gbuild and build.pl we've endured for so long.
> 
> If people are happy, I'd suggest a migration plan where first scons is
> added as a dependency, then the new version of serf is built with it (see
> i126312), then gbuild modules are ported to scons until gbuild is
> eliminated, then dmake modules are ported to scons until dmake is
> eliminated, and finally scons replaces build.pl.
> 
> Thoughts?
> 
> Damjan
> 

I am not a build architecture guru, but I will share my thoughts
anyway. As an open source project relying on volunteers, I think we
need to consider sustainability of whatever we use for building or
anything else. So this means considering what potential volunteers
have learned to use in building applications.

Our current build system is at its heart, dmake, with build.pl on
top of that to deal with integrating the non-C++ aspects into
OpenOffice and provide a means of parallel build capabilities,
unless I'm very mistaken about this. Right now, the Windows parts
are using cmake as well. What we are calling "gbuild" is using GNU
make to build. If I'm not mistaken this is a name internal to
OpenOffice and LibreOffice. And, yes, we have been slow in this
migration.

In addition to the wiki pages already referenced, we have a short
little wiki page which gives a quick overview of our current build
system:
https://wiki.openoffice.org/wiki/Build_System#Drawbacks

As far as I can discover, VisualStudio can be setup to use GNU make
for building, (see for example:
www.bordoon.com/cplusplus/VisualStudioIntegration.pdf) so I don't
see using "gbuild" as a show stopper for GUI IDEs.

My thoughts at this point are to TRY to stick with GNU Make
(gbuild), and see if we can't remove code sections which no longer
really need to be build with build.pl. I hate the use of IDLs for
header generation and would like to see us "staticize" as much as
possible.

I don't know what it would take to move our build into SCONs, or how
well it would scale for our codebase. We'd obviously need some
trials on this.

-- 
--------------------------------------------
MzK

"Though no one can go back and make a brand new start,
 anyone can start from now and make a brand new ending."
                            -- Carl Bard

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org


Mime
View raw message