mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benno Evers <bev...@mesosphere.com>
Subject Re: CMake build refactoring
Date Mon, 28 Aug 2017 09:41:28 GMT
Hi Andrew,

thanks for working on this, it looks very nice.

Is there a policy on how to handle changes to the build system before the
official deprecation of autotools? I.e., should should reviewers require
that new features are added only to autoconf, only to CMake, or to both?

Thanks,
Benno

On Sat, Aug 26, 2017 at 2:39 AM, <andrew@schwartzmeyer.com> wrote:

> Hi Zhitao,
>
> 1) I talked with Joe and confirmed that we're targeting 1.5 for this chain.
>
> 2) For Mesos modules, the included example modules are built, but I don't
> have an external example. I think the easiest approach to doing so is to
> include Mesos as a submodule and then `add_subdirectory(mesos)`, which
> would bring the target for `libmesos` into your graph to build the module
> against. I can work with anyone willing to create an example.
>
> 3) Yes, that is the plan. The JIRA issue tracking CMake is MESOS-898, and
> the last thread about the (eventual) deprecation of Autotools is here:
> https://mail-archives.apache.org/mod_mbox/mesos-dev/201703.mbox/browser
>
> Thanks,
>
> Andrew
>
>
> On 08/25/2017 2:40 pm, Zhitao Li wrote:
>
>> Hi Andrew,
>>
>> Thank you for the great work. Really looking forward to see a more
>> structured dependency graph.
>>
>> A couple of questions:
>>
>> 1) is there a target version of this work? Will the chain land on 1.5?
>>
>> 2) For people who maintains Mesos modules, is there a example build of how
>> people can arrange their build?
>>
>> 3) Is there plan to make CMake the only supported build system? If yes,
>> what's the graduation standard?
>>
>> Thanks!
>>
>> On Fri, Aug 25, 2017 at 1:00 PM, Vinod Kone <vinodkone@apache.org> wrote:
>>
>> +1
>>>
>>> On Fri, Aug 25, 2017 at 12:08 PM, <andrew@schwartzmeyer.com> wrote:
>>>
>>> > I have the Java libraries building now on both Linux and Windows
>>> (though
>>> > the initial setup on Windows is a bit annoying to get all the tools in
>>> the
>>> > right place, I'd like to follow up this chain with a `docs/cmake.md`
>>> with
>>> > all the odd little bits of information).
>>> >
>>> > I also enabled the ZooKeeper unit tests (which I needed in order to
>>> > integration test the patches that went upstream to ZooKeeper to give
>>> their
>>> > C Client library a CMake build, too).
>>> >
>>> > Furthermore, with the Java library, I've successfully run Marathon
>>> against
>>> > CMake built Mesos libraries on Linux.
>>> >
>>> > I do not have the Python libraries building yet, and there is a "Java
>>> > examples" library that is not yet built. I'll make sure to file issues
>>> to
>>> > update these.
>>> >
>>> > Joe is working to update the Mesosphere CI, I will check with him about
>>> > updating `docker-build.sh`.
>>> >
>>> > Thanks Vinod!
>>> >
>>> >
>>> > On 08/25/2017 11:59 am, Vinod Kone wrote:
>>> >
>>> >> This is great to hear. Thanks for your perseverance on making this
>>> happen
>>> >> Andy!
>>> >>
>>> >> Btw, do we build Java and Python libraries too now?
>>> >>
>>> >> Once we land the CMake changes, lets make sure to update
>>> >> `support/docker-build.sh` to make sure it is continuously being tested
>>> on
>>> >> our CI.
>>> >>
>>> >> On Fri, Aug 25, 2017 at 11:47 AM, <andrew@schwartzmeyer.com> wrote:
>>> >>
>>> >> Hello Mesos developers,
>>> >>>
>>> >>> I wanted to take the time to announce a (long) patch chain that
fixes
>>> up
>>> >>> the CMake build system. There were a lot of open bugs due to the
>>> initial
>>> >>> design (and constraints of CMake 2), and by moving to CMake 3, I
was
>>> able
>>> >>> to rewrite the CMake build in a target-based dependency graph manner
>>> (the
>>> >>> way CMake is meant to be used).
>>> >>>
>>> >>> The very end of the chain is here: https://reviews.apache.org/r/6
>>> 1753/
>>> >>>
>>> >>> Joseph Wu is currently shepherding it. That final patch includes
the
>>> what
>>> >>> and the why explanation of this patch series.
>>> >>>
>>> >>> My main guideline in this effort was MESOS-3576, which pointed out
>>> the
>>> >>> incorrectness of the link flags generated by CMake. This has been
>>> >>> resolved,
>>> >>> and moreover, a significant amount of superfluous CMake code was
>>> deleted
>>> >>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>>> >>>
>>> >>> I took a two-phase approach to this reworking. The first phase
>>> imported
>>> >>> all of our third party dependencies correctly. Instead of linking
to
>>> >>> these
>>> >>> dependencies by manually including their various include/library
>>> >>> directories and compilation/link flags, each dependency was added
to
>>> the
>>> >>> CMake graph as an imported target. All necessary information for
>>> linking
>>> >>> to
>>> >>> the dependency is recorded in its target, and used is transitively
>>> passed
>>> >>> to targets which take it as dependency by CMake. Adding a new
>>> dependency
>>> >>> is
>>> >>> now relatively simple, with many easy-to-replicate patterns available
>>> in
>>> >>> 3rdparty/CMakeLists.txt.
>>> >>>
>>> >>> An important note is that this was also done for header-only
>>> libraries,
>>> >>> including stout. CMake 3 has a notion of "interface libraries" where
>>> a
>>> >>> header-only library can be represented as a CMake target like any
>>> other
>>> >>> library. With this done correctly, linking stout to boost is as
>>> simple
>>> as
>>> >>> `target_link_libraries(stout boost)`, and linking libprocess to
stout
>>> is
>>> >>> `target_link_libraries(process stout)`, and the boost dependency
is
>>> >>> understood transitively.
>>> >>>
>>> >>> The second phase was refactoring the Mesos build itself (that is,
not
>>> the
>>> >>> third party dependencies). With all our dependencies imported
>>> properly, I
>>> >>> was able to delete the files of extraneous information such as
>>> >>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This
>>> information
>>> is
>>> >>> instead correctly stored in the aforementioned CMake dependency
>>> graph.
>>> >>>
>>> >>> With this all put together, the entirety of the required CMake code
>>> to
>>> >>> build the agent executable, on all platforms, is the following _two
>>> >>> lines_:
>>> >>>
>>> >>>     add_executable(mesos-agent main.cpp)
>>> >>>     target_link_libraries(mesos-agent PRIVATE mesos)
>>> >>>
>>> >>> All necessary link and compilation flags are parsed by CMake through
>>> the
>>> >>> dependency graph as it visits libmesos as the mesos-agent dependency.
>>> >>>
>>> >>> I keep an updated tree on GitHub here for easy testing:
>>> >>> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
>>> >>> ave/CMakeLists.txt
>>> >>>
>>> >>> As we move closer to deprecating Autotools, I wanted to ensure that
>>> the
>>> >>> replacement build system was as correct and easy-to-use as possible.
>>> As
>>> >>> with any build system (and any software engineering effort), there
is
>>> >>> always more to clean up and improve. However, I am satisfied with
the
>>> >>> result of my efforts, and I hope this build system makes your work
as
>>> >>> developers easier. If you have the time, please take a look at the
>>> the
>>> >>> patches and give it a test. Let me know know I can make it work
>>> better
>>> >>> for
>>> >>> you.
>>> >>>
>>> >>> Thank you,
>>> >>>
>>> >>> Andrew Schwartzmeyer
>>> >>>
>>> >>>
>>>
>>>


-- 
Benno Evers
Software Engineer, Mesosphere

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message