Mailing-List: contact dev-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mesos.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Fri, 25 Aug 2017 12:08:35 -0700
From: andrew@schwartzmeyer.com
To: dev@mesos.apache.org
Cc: vinodkone=apache.org@schwartzmeyer.com, Vinod Kone
 <vinodkone@apache.org>
Subject: Re: CMake build refactoring
In-Reply-To: <CANntcz72+edCjDPqxQFNohyhhGZyitNCEzG=e7ved2d1U3TRHQ@mail.gmail.com>
References: <899a9ecdea3811f7dd60b12046203a95@posteo.net>
 <CANntcz72+edCjDPqxQFNohyhhGZyitNCEzG=e7ved2d1U3TRHQ@mail.gmail.com>
Message-ID: <18703f21aeb4f45316a368463c63ba76@posteo.net>
User-Agent: Posteo Webmail
archived-at: Fri, 25 Aug 2017 19:08:42 -0000

I have the Java libraries building now on both Linux and Windows (though 
the initial setup on Windows is a bit annoying to get all the tools in 
the right place, I'd like to follow up this chain with a `docs/cmake.md` 
with all the odd little bits of information).

I also enabled the ZooKeeper unit tests (which I needed in order to 
integration test the patches that went upstream to ZooKeeper to give 
their C Client library a CMake build, too).

Furthermore, with the Java library, I've successfully run Marathon 
against CMake built Mesos libraries on Linux.

I do not have the Python libraries building yet, and there is a "Java 
examples" library that is not yet built. I'll make sure to file issues 
to update these.

Joe is working to update the Mesosphere CI, I will check with him about 
updating `docker-build.sh`.

Thanks Vinod!

On 08/25/2017 11:59 am, Vinod Kone wrote:
> This is great to hear. Thanks for your perseverance on making this 
> happen
> Andy!
> 
> Btw, do we build Java and Python libraries too now?
> 
> Once we land the CMake changes, lets make sure to update
> `support/docker-build.sh` to make sure it is continuously being tested 
> on
> our CI.
> 
> On Fri, Aug 25, 2017 at 11:47 AM, <andrew@schwartzmeyer.com> wrote:
> 
>> Hello Mesos developers,
>> 
>> I wanted to take the time to announce a (long) patch chain that fixes 
>> up
>> the CMake build system. There were a lot of open bugs due to the 
>> initial
>> design (and constraints of CMake 2), and by moving to CMake 3, I was 
>> able
>> to rewrite the CMake build in a target-based dependency graph manner 
>> (the
>> way CMake is meant to be used).
>> 
>> The very end of the chain is here: https://reviews.apache.org/r/61753/
>> 
>> Joseph Wu is currently shepherding it. That final patch includes the 
>> what
>> and the why explanation of this patch series.
>> 
>> My main guideline in this effort was MESOS-3576, which pointed out the
>> incorrectness of the link flags generated by CMake. This has been 
>> resolved,
>> and moreover, a significant amount of superfluous CMake code was 
>> deleted
>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>> 
>> I took a two-phase approach to this reworking. The first phase 
>> imported
>> all of our third party dependencies correctly. Instead of linking to 
>> these
>> dependencies by manually including their various include/library
>> directories and compilation/link flags, each dependency was added to 
>> the
>> CMake graph as an imported target. All necessary information for 
>> linking to
>> the dependency is recorded in its target, and used is transitively 
>> passed
>> to targets which take it as dependency by CMake. Adding a new 
>> dependency is
>> now relatively simple, with many easy-to-replicate patterns available 
>> in
>> 3rdparty/CMakeLists.txt.
>> 
>> An important note is that this was also done for header-only 
>> libraries,
>> including stout. CMake 3 has a notion of "interface libraries" where a
>> header-only library can be represented as a CMake target like any 
>> other
>> library. With this done correctly, linking stout to boost is as simple 
>> as
>> `target_link_libraries(stout boost)`, and linking libprocess to stout 
>> is
>> `target_link_libraries(process stout)`, and the boost dependency is
>> understood transitively.
>> 
>> The second phase was refactoring the Mesos build itself (that is, not 
>> the
>> third party dependencies). With all our dependencies imported 
>> properly, I
>> was able to delete the files of extraneous information such as
>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This information 
>> is
>> instead correctly stored in the aforementioned CMake dependency graph.
>> 
>> With this all put together, the entirety of the required CMake code to
>> build the agent executable, on all platforms, is the following _two 
>> lines_:
>> 
>>     add_executable(mesos-agent main.cpp)
>>     target_link_libraries(mesos-agent PRIVATE mesos)
>> 
>> All necessary link and compilation flags are parsed by CMake through 
>> the
>> dependency graph as it visits libmesos as the mesos-agent dependency.
>> 
>> I keep an updated tree on GitHub here for easy testing:
>> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
>> ave/CMakeLists.txt
>> 
>> As we move closer to deprecating Autotools, I wanted to ensure that 
>> the
>> replacement build system was as correct and easy-to-use as possible. 
>> As
>> with any build system (and any software engineering effort), there is
>> always more to clean up and improve. However, I am satisfied with the
>> result of my efforts, and I hope this build system makes your work as
>> developers easier. If you have the time, please take a look at the the
>> patches and give it a test. Let me know know I can make it work better 
>> for
>> you.
>> 
>> Thank you,
>> 
>> Andrew Schwartzmeyer
>>