mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cody Maloney" <>
Subject Re: Review Request 26426: Add --enable-debug flag to ./configure for controlling emission of debug information
Date Thu, 16 Oct 2014 03:52:51 GMT

> On Oct. 14, 2014, 9:06 p.m., Timothy St. Clair wrote:
> >, line 281
> > <>
> >
> >     Is there a reason you want to leave debug symbols out of optimized builds? 

> >     
> >     cmake has the pattern correct imho: 
> >     Release
> >     Debug
> >     ReleaseWithDebug
> >     
> >     A ReleaseWithDebug allows packagers, such as myself, to build w/debugsymbols
that are stripped out into a .debuginfo package which can be used by developers for tracing
"When bears attack".  Granted that it is tenuous debugging at best, but it's better then nothing.

> >     
> >     So I think we want all three modes, stripping all debug information is not really
> Cody Maloney wrote:
>     My main motivation is to shrink the size of libmesos. Yesterday sugis in #mesos had
one which was 213M. For the buildbot internally, full debian packages (which are compressed)
of mesos weign in at 165M a piece (Yes, stripping post-build would help a lot, but why build
it to begin with?). Most of this is debug info. Also, we build a bunch of different ways,
and when libmesos is as big as it is, a decent amount of time ends up being spent on disk
I/O reading / writing all the debug info when we are really just trying to ensure it builds
on all the different platforms (Not to mention storage and file size shipping things around
the network to centralized repositories).
>     The simple toggle between debug and release, removing the legacy logic gets us most
of the benefit.
>     If you have a good place to point me for what is needed to get a 'ReleaseWithDebug'
info build up and running I can definitely work on adding that as well.
> Cody Maloney wrote:
>     It is also entirely possible to specify custom CXXFLAGS + CFLAGS with this patch
to get the old "optimized debug" build. The flags set by --enable-debug (or not having it),
to get back the "optimized debug" build from before.
> Timothy St. Clair wrote:
>     I suppose rpm builds default CXXFLAGS to '-O2 -g ...', so I can buy the argument
of just making it easier on the average person.
> Ben Mahler wrote:
>     Re-opening because conflating debug and optimization under a single flag seems a
bit confusing to me.
>     In practice, we run mesos **with optimizations** (for free performance win), and
**with debugging symbols** (for meaningful backtraces and core dumps), which was previously
the default but now requires the setting of CXXFLAGS to obtain. :(
>     For development, I think by default we need debugging symbols turned on. Otherwise
most people will forget to configure with `--enable-debug` and consequently will need to recompile
everything when they encounter a backtrace or need to debug. CI jobs will encounter this issue
as well.
>     If there are use cases that merit no debugging symbols, seems nice to make that explicit
given it runs the risk of making debugging difficult.
>     Thoughts?

Note that the two were conflated under one flag previously. This just changes how we do it
to be a little more standard way of conflating them. Generally people talk about `Debug` or
`Relase` builds. Not the specific compiler flags that get switched. The general switch is
-O2 -> -O0 + -g and back (Possibly with a couple extra warning flags). Autotools projects
which implement a debug vs. release build almost always do '--enable-debug' / the default
is 'optimized' build. - that artical has some good explanations why debug
info gets problematic as programs get larger and larger, which is only going to get worse
over time. Already mesos debug info is over 100MB. As the codebase grows, this will get worse
than it already is, increasing the minimum machine specs you need to develop / compile Mesos.

Compiler implementors generally give that '-O2' should be used to generate an optimal binary.
'-g' should be used when you are planning to attach GDB and step through a program line by
line. General development generally people want nice backtraces, which is inbetween those.
See **Backtraces**. The section **Coredumps** talks about how we can give all the info in
coredumps without running full debug builds ('-g') everywhere.

**Backtraces** should generally be reasonably meaningful as long as you don't strip the binary.
Most of the symbols are still there (You just lose inlined things). You don't get file names
and line numbers, but all the functions in mesos should be uniquely named (C++'s ODR rule),
so you get out a mangled C++ name which you can demangle with `c++filt` (Shipped with GCC)
and you will get the full namespace prefixed name of the C++ function. Find the function with
your favorite text  editor search, life is happy.

Pretty much every binary which is shipped on your machine via a distribution is sent like
this. The backtraces + coredumps that are gathered come from this.

[Clang has a option -gline-tables-only](
which adds just the file name and line numbers to all functions in the debug info (Which can
be done with minimal space overhead. [Gcc has '-g1'](,
which would mean that you would get line number + file names in back traces with less overhead.

Those options would make it so what people see printed by the build bot is exactly what they
see now. I think that should be an easily accessible level, but not default build.

**Coredumps**: It is possible to make coredumps just as inspectable without including debug
information in all the shipping binaries. In fact, you generally probably don't want to run
gdb on the binary on the production host anyways. For this, what typically happens is that

1. People actually only look at the backtrace to figure out what went wrong
2. You can generate debug information which lives in seperate files from the target executable
(-gsplit-dwarf on new versions of GCC, requires a relatively new toolchain. I believe there
is a way to do it on older tooling where you split it out later, but am not familiar with
it). This debug info would generally live in a '-dbg' package. Which you can install / run
on your local machine to inspect whatever variables remain directly, as well as global state.

I don't see anywhere where this makes debugging considerably more difficult than it is already.
Yes, you can't attach a GDB to any build of mesos without thinking about it. I think most
of the time most developers aren't attaching GDB.

CI jobs I think can be easily switched over where they care. There aren't that many of them
around. I agree it is something people need to be aware of, but not a hard level of change.
Most CI systems don't let people connect in and step through the program with GDB to where
it failed a test. It just gives a backtrace. In the case of a buildbot it isn't hard to set
CXXFLAGS to just the level you want to get that ('-g1'). No major loss there with this change.

The only case this makes considerably more difficult is that if you want to attach GDB to
inspect variabels you need to have specified --enable-debug at configure time. I don't think
that is a major loss, or something which is in the standard compile/test loop of most most
Mesos contributors / developers (Although we can collect evidence if desired)

- Cody

This is an automatically generated e-mail. To reply, visit:

On Oct. 14, 2014, 11:07 p.m., Cody Maloney wrote:
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> -----------------------------------------------------------
> (Updated Oct. 14, 2014, 11:07 p.m.)
> Review request for mesos, Benjamin Hindman and Timothy St. Clair.
> Repository: mesos-git
> Description
> -------
> Reworks building mesos in "debug" vs. "release". By default, mesos is now built in release
(no debug info, optimized build). If '--enable-debug' is specified to configure, than optimization
will be turned off, and debug info will be turned on.
> This also adds a variable 'DEBUG' to the build environment, which people can use in code
to see if mesos is built with debugging to enable extra assertions / checks. For release builds
we may want to set 'NDEBUG' which removes assert()'s, but that is a seperate discussion.
> Main benefits:
> 1) Getting a build to include/exclude debug information at will is feasible. Before some
things like using clang would forcibly enable debug info in all cases
> 2) and the other binaries which get packaged up for use in distributions
shrink considerably without manually stripping post-build (Improves build time, makes packaging
> Diffs
> -----
> 2b372e06006250b5230956ef096473e98f3fa590 
> Diff:
> Testing
> -------
> Built with both --enable-debug and without, checking that the flags get passed through
> Thanks,
> Cody Maloney

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message