mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benno Evers <bev...@mesosphere.com>
Subject [Proposal] Use jemalloc as default memory allocator for Mesos
Date Fri, 18 Aug 2017 10:49:51 GMT
Hi all,

I would like to propose bundling jemalloc as a new dependency
under `3rdparty/`, and to link Mesos against this new memory
allocator by default.


# Motivation

The Mesos master and agent binaries are, ideally, very long-running
processes. This makes them susceptible to memory issues, because
even small leaks have a chance to build up over time to the point
where they become problematic.

We have seen several such issues on our internal Mesos installations,
for example https://issues.apache.org/jira/browse/MESOS-7748
or https://issues.apache.org/jira/browse/MESOS-7800.

I imagine any organization running Mesos for an extended period
of time has had its share of similar issues, so I expect this
proposal to be useful for the whole community.


# Why jemalloc?

Given that memory issues tend to be most visible after a given
process has been running for a long time, it would be great to
have the option to enable heap tracking and profiling at runtime,
without having to restart the process. (This ability could then
be connected to a Mesos endpoint, similar to how we can adjust
the log level at runtime)

The two production-quality memory allocators that have this
ability currently seem to be tcmalloc and jemalloc. Of these,
jemalloc does produce in our experience better and more
detailed statistics.


# What is the impact on users who do not need this feature?

Naturally, not every single user of Mesos will have a need
for this feature. To ensure these users would not experience serious
performance regressions as a result of this change, we conducted
a preliminary set of benchmarks whose results are collected
under https://issues.apache.org/jira/browse/MESOS-7876

It turns out that we could probably even expect a small speedup (1% - 5%)
as a nice side-effect of this change.

Users who compile Mesos themselves would of course have the option
to disable jemalloc at configuration time or replace it with their
memory allocator of choice.



I'm looking forward to hear any thoughts and comments.


Thanks,
-- 
Benno Evers
Software Engineer, Mesosphere

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message