openoffice-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andre Fischer <awf....@gmail.com>
Subject Re: [proposal] replace build.pl with a central Makefile.
Date Mon, 21 Oct 2013 08:58:14 GMT
On 20.10.2013 12:40, janI wrote:
> On 19 October 2013 19:20, Kay Schenk <kay.schenk@gmail.com> wrote:
>
>> On Fri, Oct 18, 2013 at 7:52 AM, Andre Fischer <awf.aoo@gmail.com> wrote:
>>
>>> On 18.10.2013 15:58, janI wrote:
>>>
>>>> On 18 October 2013 15:00, Andre Fischer <awf.aoo@gmail.com> wrote:
>>>>
>>>>   On 18.10.2013 14:02, janI wrote:
>>>>>   sd
>>>>>>
>>>>>> On 18 October 2013 13:36, Andre Fischer <awf.aoo@gmail.com>
wrote:
>>>>>>
>>>>>>    On 18.10.2013 11:32, janI wrote:
>>>>>>
>>>>>>>    Hi.
>>>>>>>
>>>>>>>> due to the discussion in thread "Mentor a new build system",
I have
>>>>>>>> made a
>>>>>>>> proposal for a central Makefile located in main.
>>>>>>>>
>>>>>>>>    Hi Jan,
>>>>>>>>
>>>>>>> it is great that you are going to improve this part of the build
>>>>>>> system.
>>>>>>>     But I think that we need more details about how the proposed
build
>>>>>>> system
>>>>>>> works.  Without them I can not really evaluate the proposal.
>>>>>>>
>>>>>>>    First of all, I agree with juergens remarks that this should
be
>>>>>>>
>>>>>> discussed
>>>>>> before implemented, hence the wiki page.
>>>>>>
>>>>>> Secondly this has nothing directly to do with the proposed build
>> system,
>>>>>> its a simple replacement of build.pl in the current system.
>>>>>>
>>>>>>   Yes, that is how I understood it.  I just did not know how to call
>> the
>>>>> build.pl replacement.
>>>>>
>>>>>
>>>>>
>>>>>   I know that build.pl works, but having a Makefile in main, would make
>>>>>> us
>>>>>> one step closer on being compatible with the distros. To me this
job
>> is
>>>>>> a
>>>>>> simple cleanup, not something we deadly need, but nice to have.
>>>>>>
>>>>>>
>>>>>>    Some remarks regarding the missing options:
>>>>>>
>>>>>>> --from <module>
>>>>>>>       This is one of the more important options and one that
I use
>>>>>>> frequently
>>>>>>> (also in the form --all:<module>).
>>>>>>>       Note that if you are in <moduleA> and call 'make
--from
>> <moduleB>'
>>>>>>> then
>>>>>>> all modules are built
>>>>>>>       a) which <moduleA> depends on
>>>>>>>       b) but not those that <moduleB> depends on
>>>>>>>       c) Both <moduleA> and <moduleB> are built.
>>>>>>>
>>>>>>>    I have changed the documentation.
>>>>>>>
>>>>>> I use the --all:<module> myself very often, and have changed
the
>>>>>> documentation, because it is of course supported.
>>>>>>
>>>>>> The difference is that you do the call in main, but that is a minor
>>>>>> detail
>>>>>> that can be easily corrected (have <module>/Makefile calling
>>>>>> main/Makefile.
>>>>>>
>>>>>> I have also changed documentation on --html due to juergens comments.
>>>>>>
>>>>>>   I am not sure that we understand --from and --since in the same
way
>> so
>>>>> I
>>>>> will try to explain what I think they do.
>>>>>
>>>>> Let's imagine that we have a simple project with modules A, B, C, D and
>>>>> E.
>>>>> where B depends on A, C on B, D on C, and E on D.
>>>>> A ' make all' would mean 'make E'.  The dependencies would then lead
to
>>>>> building modules A, B, C, D, E in this order.
>>>>> If I am in E and call 'make --from C' then only C, D, and E should be
>>>>> built.  A 'make --since C' would only build D and E.
>>>>>
>>>>> If I am in D and call 'make --from B' then modules B, C, and D are
>> built.
>>>>>    Call 'make --since B' to build only C and D.
>>>>> Note that 'make --from' accepts more than one module name (while 'make
>>>>> --all:<module>' does not).
>>>>> Note also that in the above case (stand in D, call 'make --from B')
>>>>> module
>>>>> A is not built, regardless of whether there are changes in A or not.
>>>>>    Whereas a simple call to make (still standing in D) would build all
>>>>> modules that D depends on, directly or indirectly.  Thus the options
>>>>> '--from' and '--since' exist to actively exclude modules from being
>>>>> built.
>>>>>
>>>>> The whole thing becomes a little bit more complicated with multiple
>>>>> options to '--from' (I never use '--since' and also don't know a valid
>>>>> use
>>>>> case so I will ignore it for now) and more complex dependencies then
in
>>>>> the
>>>>> simple example above.  Let's say that if we stand in instsetoo_native
>> and
>>>>> call 'make --from svx sfx2'.  Note that svx depends on sfx2.  This
>> would
>>>>> build svx, sfx2 and all modules that depend (directly or indirectly)
on
>>>>> svx
>>>>> OR sfx2.
>>>>>
>>>>>   got it, now I just have one problem, why would you not build the
>>>> dependent
>>>> modules, if they needed to be built, thats a scenario I dont understand.
>>>> With a central makefile, <module>/makefile will not be called so we
do
>> not
>>>> waste cpu cycles.
>>>>
>>>> With the .done files, we know when a module was last built and all
>> modules
>>>> that depend it should be rebuilt which the rule
>>>> <module>.done : <module_depend>.done
>>>>
>>>> will ensure, so If we have A -> B -> C -> D
>>>>
>>>> I go in B, and call make, then when I go in D and make, B,C,D will be
>>>> made.
>>>>
>>>> If we have A -> B -> D   C -> D
>>>> and do the same then only D will be made.
>>>>
>>>> So --from is not really saving anything ?
>>>>
>>> a) In your example you first go into B then, in a second step, into D.
>>>   The '--from' option lets you do the same (well, not really the same, but
>>> see below) just from D.
>>>
>>> b) You go first to B and call make.  This makes A, if necessary, then B.
>>>   The making of A is exactly the thing that you want to prevent with the
>>> '--from' option.  Go into D and call 'make --from B'.  A is not built.
>>>
>>> c) After the discussion with you I am not sure if we still need --from
>>> because the two reasons I know for its existence my not be relevant with
>>> the new approach.
>>>
>>> c1) With the '--from' option you can tweak the dependency rules at
>> runtime
>>> (a bit).  This allows you to exclude projects from being built when you
>>> know that that is not necessary.  But from experience I know that can
>> lead
>>> to very subtle errors.  Letting the system determine what to built is
>>> usually more reliable.
>>>
>>> c2) With build.pl a 'build --all' still builds every module on which the
>>> one you are standing in depends on.  When those modules have been built
>>> previously, then no compilation takes place.  But calling dmake for a
>>> couple of directories for close to 200 modules (when you stand in
>>> instsetoo_native) takes a lot of time (several minutes on Windows), even
>>> when no file has to be compiled.  This wasteful way of doing nothing can
>> be
>>> prevented with the --from option.
>>> However, with the new approach and its .done flag files you can determine
>>> which modules need to be built much faster.  You don't have to call dmake
>>> on directories that where already built.  Hm, but this again, does only
>>> work if your .done files have dependencies on all relevant source files.
>>> That is something that is missing at least from my script.
>>>
>>>
>>> So, reasons for the existence of '--from' are a result of old/slow
>>> computers, slow files systems (still valid on Windows), missing global
>>> dependencies (which we now have for gbuild) and impatient developers.
>>
>> Looking at what Jan has proposed and this discussion so far, I think we
>> should make some attempt at moving in this direction -- using standard
>> makefiles.

Yes, agreed.

>> Given this discussion, I do understand the additional flexibility of
>> build.pl, but I also wonder if the flexibility is worth it considering the
>> IDE tools new developers are likely to be familiar with.
>>
>> A complete build will take a while no matter what is used, depending on
>> your situation.

A complete build is the easiest of the tasks of a build system.
The hard part is to handle the development stage where single file 
changes might require whole modules to be rebuilt.

>>   I think this proposal has a lot of merit.

I agree.

> Thx for the kind words.
>
> I have updated the wiki page, to reflect the specific make calls.
>
> and R1533869 shows my first version of the central Makefile
>
> https://svn.apache.org/repos/asf/openoffice/branches/capstone2013/main/Makefile
>
> you will see it contains the intermodule dependencies and nothing more. The
> setup in every module is kept in a Makefile in the respective module, so we
> have a clear seperation of module and intermodule dependencies.

The gbuild modules already have files called 'Makefile' top-level. Are 
you going to reorganize this and the prj/ and util/ directories?

-Andre

>
> hope you like it.
> rgds
> jan I.
>
>
>
>>
>>
>>
>>
>>>
>>>>   While this is easy to do with eg Perl I am not sure how to handle this
>>>>> with just a Makefile.  The straightforward approach with handling
>>>>> <module>.done files does not work.  And that is one of the reasons
why
>> I
>>>>> don't think that (GNU) makefiles are a good solution for any problem.
>>>>>   Most
>>>>> of us are used to program object oriented/imperative.  Makefiles
>> require
>>>>> a
>>>>> declarative approach. Maybe the use of Perl is not such a bad idea.
>>>>>   Maybe
>>>>> it would be better to reimplement build.pl with a lot fewer options
>> and
>>>>> with better readable code.
>>>>>
>>>>>   I agree that makefiles are nowhere near a good solution to many of
>> these
>>>> problems, but its like windows, I dont like it, but everybody uses it.
>>>>
>>>> We could easily write a new build.pl, that also took care of the local
>>>> makefiles, but our build system would not be in the mainstream, and e.g.
>>>> the distros would not like to integrate AOO.
>>>>
>>>> I have over the last years followed research in building systems, and
>>>> there
>>>> are (sadly enough) nobody that tries a real object oriented aproach.
>> Also
>>>> if you look at packages like visual studio, QT, eclipse they all use the
>>>> principle of makefiles with declarative approach.
>>>>
>>>> So my simple question is, do we want to approach the main road
>> (makefiles
>>>> for unix, visual studio for windows/mac) or do we want to have better
>> but
>>>> non standard system.
>>>>
>>> Good analysis.  Maybe I should answer with Faust: "Zwei Seelen wohnen,
>>> ach! in meiner Brust" (two souls alas! are dwelling in my breast).  The
>>> pragmatist says to use the make.  It is good enough for others, it is
>> good
>>> enough for use.
>>> On the other hand, when I start a new project I usually start with the
>>> question of what are the best tools for the problem at hand. And make
>> does
>>> not seem to be the first or the best answer.  Look at our dmake or gbuild
>>> system.  Both don't use make in a standard way. gmake even defines its
>> own
>>> language, object oriented and imperative, on top of the GNU make macros.
>>>   That is, for me, an act of desperation.
>>> I have made experiments with an alternative approach, a domain specific
>>> language somewhat similar to Java.  I personally like this approach
>> because
>>> a) it uses the paradigm that I already use when writing C++ code. That
>>> means that I can apply my existing knowledge to the build process and
>> that
>>> I don't have to remember all the tricks and pitfalls of makefiles.
>>> b) as expected it was much easier to handle file dependencies and
>> parallel
>>> processing of build jobs in Java than adding object orientation and
>>> imperative control flow to makefiles.
>>>
>>> If I had the time and if I would be the one working on it then I would
>>> prefer an non-Makefile approach.  But maybe I am just suffering too much
>>> from one of the 'three great virtues of a programmer': hubris.
>>>
>>> You are the one who leads the build project changes, so you have to
>> decide
>>> which approach to use.  I am not trying to make your life harder (than
>>> necessary), I am only trying to point out some of the pitfalls that I
>> have
>>> encountered in the past.  And to prevent you from removing features that
>> I
>>> use :-)
>>>
>>>
>>>
>>>
>>>> rgds
>>>> jan I.
>>>>
>>>> Ps. its always refreshing to discuss with you, you often have an
>>>> alternative approach, which is not just a dream but doable.
>>>>
>>> Thanks. That makes two of us.
>>>
>>> Have a nice weekend,
>>> Andre
>>>
>>>
>>>
>>>>   -Andre
>>>>>
>>>>>     --prepare
>>>>>>>       Also one option that is important for our every day work.
 Use
>>>>>>> case:
>>>>>>> You make changes in <module> and are not sure if these
changes are
>>>>>>> compatible/incompatible.  To be on the safe side you discard
the
>> output
>>>>>>> of
>>>>>>> all depending modules.  To save time you keep the output of all
other
>>>>>>> modules.
>>>>>>>
>>>>>>>       Often used together with '--from' like 'make --prepare
--from
>>>>>>> svx' to
>>>>>>> prepare a build after making changes in svx.
>>>>>>>
>>>>>>>    Documentation changed, funny thing is that svx does not clear
>>>>>>> correctly
>>>>>>>
>>>>>> on
>>>>>> my ubuntu build.
>>>>>>
>>>>>>
>>>>>>    --since <module>
>>>>>>
>>>>>>>       A variant of '--from'.  The only difference is that <module>
>>>>>>> itself
>>>>>>> is
>>>>>>> not built.
>>>>>>>
>>>>>>>       If your proposed approach is similar to what my script
produces
>>>>>>> then
>>>>>>> it
>>>>>>> is not too difficult to support --from/--since.  I made some
>>>>>>> experiments
>>>>>>> in
>>>>>>> this direction but was to lazy to finish them.
>>>>>>>
>>>>>>>    My approach is very similar, but I failed to see how --since
is
>>>>>>>
>>>>>> supported.
>>>>>> And question is if its real important.
>>>>>>
>>>>>>
>>>>>>    --job
>>>>>>
>>>>>>> --pre_job
>>>>>>> --post_job
>>>>>>>      These are sometimes handy to run a non-standard command
for all
>>>>>>> modules.
>>>>>>>
>>>>>>>    I have added them, they are by the way a good example why
we need a
>>>>>>>
>>>>>> discussion I have never used them.
>>>>>>
>>>>>> However maybe the real discussion is "do we want to replace build
and
>>>>>> have
>>>>>> a main/Makefile instead?"
>>>>>>
>>>>>>
>>>>>>
>>>>>>    - I have not used the rest of the unsupported options and would
not
>>>>>> miss
>>>>>>
>>>>>>> them.  Others may have other sets of options that are important
to
>>>>>>> them.
>>>>>>>
>>>>>>>
>>>>>>> Some general remarks:
>>>>>>>
>>>>>>> - Why keep one makefile per module?  Why not put all the inter-module
>>>>>>> dependencies into one file (like my script does)?
>>>>>>>
>>>>>>>    Ups, I did not explain that correctly, I propose 1 Makefile
>>>>>>>
>>>>>> "main/Makefile"
>>>>>> with all inter-module and 1 Makefile "<module>/Makefile" that
today
>> just
>>>>>> will call the old makefiles as described in prj/build.lst
>>>>>>
>>>>>> - Why not use the oportunity to move (a part of) the build environment
>>>>>> out
>>>>>>
>>>>>>   of the way to, say, build/ ?
>>>>>>>    You have guessed my next step.
>>>>>>>
>>>>>>    - How are dependencies between modules handled (just the manual
>>>>>>
>>>>>>> dependencies from prj/build.lst or also the file dependencies
>>>>>>> introduced
>>>>>>> by
>>>>>>> gmake).
>>>>>>>
>>>>>>>    See doc. on --from. Its done with <module>.done files
>>>>>>>
>>>>>>    - How is the output of the individual calls to dmake or GNU make
>>>>>>
>>>>>>> handled/made accessible.  Ie. if there is a build error, how
can I
>> look
>>>>>>> up
>>>>>>> the corresponding build output?
>>>>>>>
>>>>>>>    see doc. script make_log
>>>>>>>
>>>>>>    - Are the gmake makefiles included (run in the same process) or
is
>> GNU
>>>>>>> make started for them it its own process?
>>>>>>>
>>>>>>>    For a start they would be called (own process), but its something
>>>>>>> where
>>>>>>>
>>>>>> I
>>>>>> have no strong opinions.
>>>>>>
>>>>>> Please (just to be sure), this proposal has nothing to do with the
>>>>>> students
>>>>>> work, its simply because I saw a positive discussion on removing
>>>>>> build.pl
>>>>>> ,
>>>>>> and spent a couple of hours looking at it. If there is a preference
>> not
>>>>>> to
>>>>>> remove build.pl I will simply forget it.
>>>>>>
>>>>>> rgds
>>>>>> jan I.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>   Regards,
>>>>>>> Andre
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    It has been roughly tested it, thanks to a clever utility
from
>> andre.
>>>>>>>> As discussed build.pl contains a lot of options, which need
to be
>>>>>>>> considered in a makefile.
>>>>>>>>
>>>>>>>> My suggestion is on
>>>>>>>> http://wiki.openoffice.org/******wiki/Build_System_Analysis:**<
>> http://wiki.openoffice.org/****wiki/Build_System_Analysis:**>
>>>>>>>> **<http://wiki.openoffice.org/****wiki/Build_System_Analysis:**<
>> http://wiki.openoffice.org/**wiki/Build_System_Analysis:**>
>>>>>>>> build.pl_versus_makefile<http:****//wiki.openoffice.org/wiki/****<
>> http://wiki.openoffice.org/wiki/**>
>>>>>>>> Build_System_Analysis:build.****pl_versus_makefile<http://**
>>>>>>>> wiki.openoffice.org/wiki/**Build_System_Analysis:build.**
>>>>>>>> pl_versus_makefile<
>> http://wiki.openoffice.org/wiki/Build_System_Analysis:build.pl_versus_makefile
>>>>>>>> Please feel free to edit/comment on the page. I have reduced
to
>>>>>>>> options
>>>>>>>> a
>>>>>>>> lot, and some of them might be in use.
>>>>>>>>
>>>>>>>> thanks in advance for your comments.
>>>>>>>>
>>>>>>>>
>>>>>>>>    ------------------------------******--------------------------**
>>>>>>>> --**
>>>>>>>>
>>>>>>> --**---------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.****a**pache.org<
>>>>>>> http://apache.org**>
>>>>>>> <dev-unsubscribe@**openoffice.**apache.org<
>> http://openoffice.apache.org>
>>>>>>> <dev-unsubscribe@**openoffice.apache.org<
>> dev-unsubscribe@openoffice.apache.org>
>>>>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>   ------------------------------****----------------------------**
>>>>> --**---------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**a**pache.org<
>> http://apache.org>
>>>>> <dev-unsubscribe@**openoffice.apache.org<
>> dev-unsubscribe@openoffice.apache.org>
>>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>>
>>>>>
>>>>>
>>> ------------------------------**------------------------------**---------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<
>> dev-unsubscribe@openoffice.apache.org>
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>>
>>
>> --
>>
>> -------------------------------------------------------------------------------------------------
>> MzK
>>
>> "Truth is stranger than fiction, but it is because Fiction is obliged
>>   to stick to possibilities. Truth isn't."
>>                               -- "Following the Equator", Mark Twain
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org


Mime
View raw message