mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Coffler <Jeff.Coff...@microsoft.com.INVALID>
Subject RE: CMake build refactoring
Date Mon, 28 Aug 2017 22:04:07 GMT
Looks like Andy didn't answer this, so I will.

Currently Linux builds still use autotools but can use cmake. Buildbot uses autotools for
Linux. Windows builds exclusively uses cmake.

End result: For now, reviewers should insure that changes which affect files (new source files,
new header files for autotools) should work in both autoconf and cmake. This will continue
until final deprecation of autotools. I'm unsure when that will be, but it'll be a while.

If you don't do that, then you will break some build (and some buildbot).

/Jeff

-----Original Message-----
From: Benno Evers [mailto:bevers@mesosphere.com] 
Sent: Monday, August 28, 2017 2:41 AM
To: dev@mesos.apache.org
Subject: Re: CMake build refactoring

Hi Andrew,

thanks for working on this, it looks very nice.

Is there a policy on how to handle changes to the build system before the official deprecation
of autotools? I.e., should should reviewers require that new features are added only to autoconf,
only to CMake, or to both?

Thanks,
Benno

On Sat, Aug 26, 2017 at 2:39 AM, <andrew@schwartzmeyer.com> wrote:

> Hi Zhitao,
>
> 1) I talked with Joe and confirmed that we're targeting 1.5 for this chain.
>
> 2) For Mesos modules, the included example modules are built, but I 
> don't have an external example. I think the easiest approach to doing 
> so is to include Mesos as a submodule and then 
> `add_subdirectory(mesos)`, which would bring the target for `libmesos` 
> into your graph to build the module against. I can work with anyone willing to create
an example.
>
> 3) Yes, that is the plan. The JIRA issue tracking CMake is MESOS-898, 
> and the last thread about the (eventual) deprecation of Autotools is here:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-
> archives.apache.org%2Fmod_mbox%2Fmesos-dev%2F201703.mbox%2Fbrowser&dat
> a=02%7C01%7CJeff.Coffler%40microsoft.com%7Ceaf570e831614a3bac2608d4edf
> 8fc46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636395101029885531&
> sdata=vqqfe%2BQJqOoh6ZMp15W4zHO0iNj3WFYM7nkRU6ovgxE%3D&reserved=0
>
> Thanks,
>
> Andrew
>
>
> On 08/25/2017 2:40 pm, Zhitao Li wrote:
>
>> Hi Andrew,
>>
>> Thank you for the great work. Really looking forward to see a more 
>> structured dependency graph.
>>
>> A couple of questions:
>>
>> 1) is there a target version of this work? Will the chain land on 1.5?
>>
>> 2) For people who maintains Mesos modules, is there a example build 
>> of how people can arrange their build?
>>
>> 3) Is there plan to make CMake the only supported build system? If 
>> yes, what's the graduation standard?
>>
>> Thanks!
>>
>> On Fri, Aug 25, 2017 at 1:00 PM, Vinod Kone <vinodkone@apache.org> wrote:
>>
>> +1
>>>
>>> On Fri, Aug 25, 2017 at 12:08 PM, <andrew@schwartzmeyer.com> wrote:
>>>
>>> > I have the Java libraries building now on both Linux and Windows
>>> (though
>>> > the initial setup on Windows is a bit annoying to get all the 
>>> > tools in
>>> the
>>> > right place, I'd like to follow up this chain with a 
>>> > `docs/cmake.md`
>>> with
>>> > all the odd little bits of information).
>>> >
>>> > I also enabled the ZooKeeper unit tests (which I needed in order 
>>> > to integration test the patches that went upstream to ZooKeeper to 
>>> > give
>>> their
>>> > C Client library a CMake build, too).
>>> >
>>> > Furthermore, with the Java library, I've successfully run Marathon
>>> against
>>> > CMake built Mesos libraries on Linux.
>>> >
>>> > I do not have the Python libraries building yet, and there is a 
>>> > "Java examples" library that is not yet built. I'll make sure to 
>>> > file issues
>>> to
>>> > update these.
>>> >
>>> > Joe is working to update the Mesosphere CI, I will check with him 
>>> > about updating `docker-build.sh`.
>>> >
>>> > Thanks Vinod!
>>> >
>>> >
>>> > On 08/25/2017 11:59 am, Vinod Kone wrote:
>>> >
>>> >> This is great to hear. Thanks for your perseverance on making 
>>> >> this
>>> happen
>>> >> Andy!
>>> >>
>>> >> Btw, do we build Java and Python libraries too now?
>>> >>
>>> >> Once we land the CMake changes, lets make sure to update 
>>> >> `support/docker-build.sh` to make sure it is continuously being 
>>> >> tested
>>> on
>>> >> our CI.
>>> >>
>>> >> On Fri, Aug 25, 2017 at 11:47 AM, <andrew@schwartzmeyer.com> wrote:
>>> >>
>>> >> Hello Mesos developers,
>>> >>>
>>> >>> I wanted to take the time to announce a (long) patch chain that

>>> >>> fixes
>>> up
>>> >>> the CMake build system. There were a lot of open bugs due to the
>>> initial
>>> >>> design (and constraints of CMake 2), and by moving to CMake 3, I

>>> >>> was
>>> able
>>> >>> to rewrite the CMake build in a target-based dependency graph 
>>> >>> manner
>>> (the
>>> >>> way CMake is meant to be used).
>>> >>>
>>> >>> The very end of the chain is here: 
>>> >>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>> >>> Freviews.apache.org%2Fr%2F6&data=02%7C01%7CJeff.Coffler%40micros
>>> >>> oft.com%7Ceaf570e831614a3bac2608d4edf8fc46%7C72f988bf86f141af91a
>>> >>> b2d7cd011db47%7C1%7C0%7C636395101029885531&sdata=V6dxdCiuNSlhWQL
>>> >>> hkE2k1R3coDGVBPWz1p1XymuVxMM%3D&reserved=0
>>> 1753/
>>> >>>
>>> >>> Joseph Wu is currently shepherding it. That final patch includes

>>> >>> the
>>> what
>>> >>> and the why explanation of this patch series.
>>> >>>
>>> >>> My main guideline in this effort was MESOS-3576, which pointed 
>>> >>> out
>>> the
>>> >>> incorrectness of the link flags generated by CMake. This has 
>>> >>> been resolved, and moreover, a significant amount of superfluous

>>> >>> CMake code was
>>> deleted
>>> >>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>>> >>>
>>> >>> I took a two-phase approach to this reworking. The first phase
>>> imported
>>> >>> all of our third party dependencies correctly. Instead of 
>>> >>> linking to these dependencies by manually including their 
>>> >>> various include/library directories and compilation/link flags,

>>> >>> each dependency was added to
>>> the
>>> >>> CMake graph as an imported target. All necessary information for
>>> linking
>>> >>> to
>>> >>> the dependency is recorded in its target, and used is 
>>> >>> transitively
>>> passed
>>> >>> to targets which take it as dependency by CMake. Adding a new
>>> dependency
>>> >>> is
>>> >>> now relatively simple, with many easy-to-replicate patterns 
>>> >>> available
>>> in
>>> >>> 3rdparty/CMakeLists.txt.
>>> >>>
>>> >>> An important note is that this was also done for header-only
>>> libraries,
>>> >>> including stout. CMake 3 has a notion of "interface libraries" 
>>> >>> where
>>> a
>>> >>> header-only library can be represented as a CMake target like 
>>> >>> any
>>> other
>>> >>> library. With this done correctly, linking stout to boost is as
>>> simple
>>> as
>>> >>> `target_link_libraries(stout boost)`, and linking libprocess to

>>> >>> stout
>>> is
>>> >>> `target_link_libraries(process stout)`, and the boost dependency

>>> >>> is understood transitively.
>>> >>>
>>> >>> The second phase was refactoring the Mesos build itself (that 
>>> >>> is, not
>>> the
>>> >>> third party dependencies). With all our dependencies imported
>>> properly, I
>>> >>> was able to delete the files of extraneous information such as 
>>> >>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This
>>> information
>>> is
>>> >>> instead correctly stored in the aforementioned CMake dependency
>>> graph.
>>> >>>
>>> >>> With this all put together, the entirety of the required CMake 
>>> >>> code
>>> to
>>> >>> build the agent executable, on all platforms, is the following 
>>> >>> _two
>>> >>> lines_:
>>> >>>
>>> >>>     add_executable(mesos-agent main.cpp)
>>> >>>     target_link_libraries(mesos-agent PRIVATE mesos)
>>> >>>
>>> >>> All necessary link and compilation flags are parsed by CMake 
>>> >>> through
>>> the
>>> >>> dependency graph as it visits libmesos as the mesos-agent dependency.
>>> >>>
>>> >>> I keep an updated tree on GitHub here for easy testing:
>>> >>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>> >>> Fgithub.com%2Fandschwa%2Fmesos%2Fblob%2Fcmake-refactor%2Fsrc%2Fs
>>> >>> l&data=02%7C01%7CJeff.Coffler%40microsoft.com%7Ceaf570e831614a3b
>>> >>> ac2608d4edf8fc46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63
>>> >>> 6395101029885531&sdata=rNiY2lyXnOTDv2TvKdhHjBIYcJSXqebZTfTsqWasR
>>> >>> sE%3D&reserved=0
>>> >>> ave/CMakeLists.txt
>>> >>>
>>> >>> As we move closer to deprecating Autotools, I wanted to ensure 
>>> >>> that
>>> the
>>> >>> replacement build system was as correct and easy-to-use as possible.
>>> As
>>> >>> with any build system (and any software engineering effort), 
>>> >>> there is always more to clean up and improve. However, I am 
>>> >>> satisfied with the result of my efforts, and I hope this build 
>>> >>> system makes your work as developers easier. If you have the 
>>> >>> time, please take a look at the
>>> the
>>> >>> patches and give it a test. Let me know know I can make it work
>>> better
>>> >>> for
>>> >>> you.
>>> >>>
>>> >>> Thank you,
>>> >>>
>>> >>> Andrew Schwartzmeyer
>>> >>>
>>> >>>
>>>
>>>


--
Benno Evers
Software Engineer, Mesosphere
Mime
View raw message