Mailing-List: contact dev-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mesos.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Fri, 25 Aug 2017 17:39:57 -0700
From: andrew@schwartzmeyer.com
To: dev@mesos.apache.org
Cc: vinodkone=apache.org@schwartzmeyer.com, Vinod Kone
 <vinodkone@apache.org>, zhitaoli.cs=gmail.com@schwartzmeyer.com, Zhitao Li
 <zhitaoli.cs@gmail.com>
Subject: Re: CMake build refactoring
In-Reply-To: <CANMwFmWggWSFdH7JfhXMBPE62jLOS_=Vg1=nZ_nPLmnknCbVwg@mail.gmail.com>
References: <899a9ecdea3811f7dd60b12046203a95@posteo.net>
 <CANntcz72+edCjDPqxQFNohyhhGZyitNCEzG=e7ved2d1U3TRHQ@mail.gmail.com>
 <18703f21aeb4f45316a368463c63ba76@posteo.net>
 <CANntcz4cxDj3bZtZQ=7dj-EaWHOcCpW=Sci5sx=wiK7yMc_uCQ@mail.gmail.com>
 <CANMwFmWggWSFdH7JfhXMBPE62jLOS_=Vg1=nZ_nPLmnknCbVwg@mail.gmail.com>
Message-ID: <cd60841ab7657808c5ecb7a59845e944@posteo.net>
User-Agent: Posteo Webmail
archived-at: Sat, 26 Aug 2017 00:40:05 -0000

Hi Zhitao,

1) I talked with Joe and confirmed that we're targeting 1.5 for this 
chain.

2) For Mesos modules, the included example modules are built, but I 
don't have an external example. I think the easiest approach to doing so 
is to include Mesos as a submodule and then `add_subdirectory(mesos)`, 
which would bring the target for `libmesos` into your graph to build the 
module against. I can work with anyone willing to create an example.

3) Yes, that is the plan. The JIRA issue tracking CMake is MESOS-898, 
and the last thread about the (eventual) deprecation of Autotools is 
here: 
https://mail-archives.apache.org/mod_mbox/mesos-dev/201703.mbox/browser

Thanks,

Andrew

On 08/25/2017 2:40 pm, Zhitao Li wrote:
> Hi Andrew,
> 
> Thank you for the great work. Really looking forward to see a more
> structured dependency graph.
> 
> A couple of questions:
> 
> 1) is there a target version of this work? Will the chain land on 1.5?
> 
> 2) For people who maintains Mesos modules, is there a example build of 
> how
> people can arrange their build?
> 
> 3) Is there plan to make CMake the only supported build system? If yes,
> what's the graduation standard?
> 
> Thanks!
> 
> On Fri, Aug 25, 2017 at 1:00 PM, Vinod Kone <vinodkone@apache.org> 
> wrote:
> 
>> +1
>> 
>> On Fri, Aug 25, 2017 at 12:08 PM, <andrew@schwartzmeyer.com> wrote:
>> 
>> > I have the Java libraries building now on both Linux and Windows (though
>> > the initial setup on Windows is a bit annoying to get all the tools in
>> the
>> > right place, I'd like to follow up this chain with a `docs/cmake.md`
>> with
>> > all the odd little bits of information).
>> >
>> > I also enabled the ZooKeeper unit tests (which I needed in order to
>> > integration test the patches that went upstream to ZooKeeper to give
>> their
>> > C Client library a CMake build, too).
>> >
>> > Furthermore, with the Java library, I've successfully run Marathon
>> against
>> > CMake built Mesos libraries on Linux.
>> >
>> > I do not have the Python libraries building yet, and there is a "Java
>> > examples" library that is not yet built. I'll make sure to file issues to
>> > update these.
>> >
>> > Joe is working to update the Mesosphere CI, I will check with him about
>> > updating `docker-build.sh`.
>> >
>> > Thanks Vinod!
>> >
>> >
>> > On 08/25/2017 11:59 am, Vinod Kone wrote:
>> >
>> >> This is great to hear. Thanks for your perseverance on making this
>> happen
>> >> Andy!
>> >>
>> >> Btw, do we build Java and Python libraries too now?
>> >>
>> >> Once we land the CMake changes, lets make sure to update
>> >> `support/docker-build.sh` to make sure it is continuously being tested
>> on
>> >> our CI.
>> >>
>> >> On Fri, Aug 25, 2017 at 11:47 AM, <andrew@schwartzmeyer.com> wrote:
>> >>
>> >> Hello Mesos developers,
>> >>>
>> >>> I wanted to take the time to announce a (long) patch chain that fixes
>> up
>> >>> the CMake build system. There were a lot of open bugs due to the
>> initial
>> >>> design (and constraints of CMake 2), and by moving to CMake 3, I was
>> able
>> >>> to rewrite the CMake build in a target-based dependency graph manner
>> (the
>> >>> way CMake is meant to be used).
>> >>>
>> >>> The very end of the chain is here: https://reviews.apache.org/r/61753/
>> >>>
>> >>> Joseph Wu is currently shepherding it. That final patch includes the
>> what
>> >>> and the why explanation of this patch series.
>> >>>
>> >>> My main guideline in this effort was MESOS-3576, which pointed out the
>> >>> incorrectness of the link flags generated by CMake. This has been
>> >>> resolved,
>> >>> and moreover, a significant amount of superfluous CMake code was
>> deleted
>> >>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>> >>>
>> >>> I took a two-phase approach to this reworking. The first phase imported
>> >>> all of our third party dependencies correctly. Instead of linking to
>> >>> these
>> >>> dependencies by manually including their various include/library
>> >>> directories and compilation/link flags, each dependency was added to
>> the
>> >>> CMake graph as an imported target. All necessary information for
>> linking
>> >>> to
>> >>> the dependency is recorded in its target, and used is transitively
>> passed
>> >>> to targets which take it as dependency by CMake. Adding a new
>> dependency
>> >>> is
>> >>> now relatively simple, with many easy-to-replicate patterns available
>> in
>> >>> 3rdparty/CMakeLists.txt.
>> >>>
>> >>> An important note is that this was also done for header-only libraries,
>> >>> including stout. CMake 3 has a notion of "interface libraries" where a
>> >>> header-only library can be represented as a CMake target like any other
>> >>> library. With this done correctly, linking stout to boost is as simple
>> as
>> >>> `target_link_libraries(stout boost)`, and linking libprocess to stout
>> is
>> >>> `target_link_libraries(process stout)`, and the boost dependency is
>> >>> understood transitively.
>> >>>
>> >>> The second phase was refactoring the Mesos build itself (that is, not
>> the
>> >>> third party dependencies). With all our dependencies imported
>> properly, I
>> >>> was able to delete the files of extraneous information such as
>> >>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This information
>> is
>> >>> instead correctly stored in the aforementioned CMake dependency graph.
>> >>>
>> >>> With this all put together, the entirety of the required CMake code to
>> >>> build the agent executable, on all platforms, is the following _two
>> >>> lines_:
>> >>>
>> >>>     add_executable(mesos-agent main.cpp)
>> >>>     target_link_libraries(mesos-agent PRIVATE mesos)
>> >>>
>> >>> All necessary link and compilation flags are parsed by CMake through
>> the
>> >>> dependency graph as it visits libmesos as the mesos-agent dependency.
>> >>>
>> >>> I keep an updated tree on GitHub here for easy testing:
>> >>> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
>> >>> ave/CMakeLists.txt
>> >>>
>> >>> As we move closer to deprecating Autotools, I wanted to ensure that the
>> >>> replacement build system was as correct and easy-to-use as possible. As
>> >>> with any build system (and any software engineering effort), there is
>> >>> always more to clean up and improve. However, I am satisfied with the
>> >>> result of my efforts, and I hope this build system makes your work as
>> >>> developers easier. If you have the time, please take a look at the the
>> >>> patches and give it a test. Let me know know I can make it work better
>> >>> for
>> >>> you.
>> >>>
>> >>> Thank you,
>> >>>
>> >>> Andrew Schwartzmeyer
>> >>>
>> >>>
>>