mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Coffler <Jeff.Coff...@microsoft.com.INVALID>
Subject RE: [Proposal] Use jemalloc as default memory allocator for Mesos
Date Wed, 23 Aug 2017 16:16:21 GMT
Hi Benno,

What's the availability of both jemalloc and tcmalloc on the Windows platform? Do the products
work there properly?

There are solutions that I know work on Windows (from past work I've done). I'm unsure about
either jemalloc and tcmalloc, however.

Thanks,

/Jeff

-----Original Message-----
From: Benno Evers [mailto:bevers@mesosphere.com] 
Sent: Tuesday, August 22, 2017 3:16 AM
To: dev@mesos.apache.org
Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos

Hi Alexander,

in general, jemalloc and tcmalloc are very similar, and seem to be taking ideas from each
other (in fact the jeprof executable started as a copy of pprof and there are still references
the pprof documentation in some
comments)

From what I've seen, the main difference is that the profiling seems better-suited to multi-threaded
programs, in particular the profile file format includes per-thread memory statistics and
the profiling features can be turned on and off individually per thread. From an API perspective,
all settings can be accessed by the mallctl() function, while it seems that tcmalloc requires
some options to be set by environment variable ( https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgperftools.github.io%2Fgperftools%2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636389937852256730&sdata=IQeb2%2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0).
Finally, I also found the documentation to be more thorough.

But again, the two are very similar, so I think the main decision here isn't whether to choose
jemalloc or tcmalloc but whether to switch to a custom memory allocator that has support for
profiling heap memory usage.


On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas <alexander@mesosphere.io>
wrote:

> Hi Benno,
>
> This does sound like a great addition to Mesos. Can you however 
> explain how jemalloc is better than tcmalloc? I think that for such 
> important change, we probably need some more information.
>
> Your comment in MESOS-7876 mentions that we already have tcmalloc 
> since it is part of gperftools, so I would like to have a whole 
> picture of the advantages and disadvantages of both options.
>
> Alexander Rojas
> alexander@mesosphere.io
>
>
>
>
> > On 18. Aug 2017, at 12:49, Benno Evers <bevers@mesosphere.com> wrote:
> >
> > Hi all,
> >
> > I would like to propose bundling jemalloc as a new dependency under 
> > `3rdparty/`, and to link Mesos against this new memory allocator by 
> > default.
> >
> >
> > # Motivation
> >
> > The Mesos master and agent binaries are, ideally, very long-running 
> > processes. This makes them susceptible to memory issues, because 
> > even small leaks have a chance to build up over time to the point 
> > where they become problematic.
> >
> > We have seen several such issues on our internal Mesos 
> > installations, for example 
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
> > ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7748&data=02%7C01%7CJeff.Coff
> > ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f
> > 141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=L016YGyEkK5
> > 0WtvhgSNS%2FT5ntkkd9qINorRI2Utp5lk%3D&reserved=0
> > or https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMESOS-7800&data=02%7C01%7CJeff.Coffler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=IrzDO6o1VL9a8eGJIW3jKbWXk6U4fHFn3Xbn4po1r3c%3D&reserved=0.
> >
> > I imagine any organization running Mesos for an extended period of 
> > time has had its share of similar issues, so I expect this proposal 
> > to be useful for the whole community.
> >
> >
> > # Why jemalloc?
> >
> > Given that memory issues tend to be most visible after a given 
> > process has been running for a long time, it would be great to have 
> > the option to enable heap tracking and profiling at runtime, without 
> > having to restart the process. (This ability could then be connected 
> > to a Mesos endpoint, similar to how we can adjust the log level at 
> > runtime)
> >
> > The two production-quality memory allocators that have this ability 
> > currently seem to be tcmalloc and jemalloc. Of these, jemalloc does 
> > produce in our experience better and more detailed statistics.
> >
> >
> > # What is the impact on users who do not need this feature?
> >
> > Naturally, not every single user of Mesos will have a need for this 
> > feature. To ensure these users would not experience serious 
> > performance regressions as a result of this change, we conducted a 
> > preliminary set of benchmarks whose results are collected under 
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
> > ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7876&data=02%7C01%7CJeff.Coff
> > ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f
> > 141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=RsZcAGuFm%2
> > Bw2PPLgMql%2B9vVgkFQrZZFJYdPGcBODsCU%3D&reserved=0
> >
> > It turns out that we could probably even expect a small speedup (1% 
> > - 5%) as a nice side-effect of this change.
> >
> > Users who compile Mesos themselves would of course have the option 
> > to disable jemalloc at configuration time or replace it with their 
> > memory allocator of choice.
> >
> >
> >
> > I'm looking forward to hear any thoughts and comments.
> >
> >
> > Thanks,
> > --
> > Benno Evers
> > Software Engineer, Mesosphere
>
>


--
Benno Evers
Software Engineer, Mesosphere
Mime
View raw message