mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Toenshoff <toensh...@me.com>
Subject Re: [Proposal] Use jemalloc as default memory allocator for Mesos
Date Thu, 31 Aug 2017 01:32:18 GMT
Ow, I forgot my actual vote :)

+1 for using jemalloc
+1 for making it non bundled
+1 for using it by default if the configuration phase locates it

> On Aug 25, 2017, at 3:28 PM, Benno Evers <bevers@mesosphere.com> wrote:
> 
> Hi Jeff,
> 
> from looking around on the internet, it seems like Firefox builds with
> jemalloc on all platforms, and I've also seen reports of people
> successfully using tcmalloc heap profiling on windows. I'm afraid I don't
> currently have a Windows machine with development environment set up, so I
> can't provide direct user experience.
> 
> In the worst case, we would have to disable jemalloc-support on windows,
> but I hope it won't be necessary.
> 
> Since you probably have more experience with memory management on windows,
> is there a reason to suspect that it should or shouldn't work?
> 
> Best regards,
> Benno
> 
> On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler <
> Jeff.Coffler@microsoft.com.invalid> wrote:
> 
>> Hi Benno,
>> 
>> What's the availability of both jemalloc and tcmalloc on the Windows
>> platform? Do the products work there properly?
>> 
>> There are solutions that I know work on Windows (from past work I've
>> done). I'm unsure about either jemalloc and tcmalloc, however.
>> 
>> Thanks,
>> 
>> /Jeff
>> 
>> -----Original Message-----
>> From: Benno Evers [mailto:bevers@mesosphere.com]
>> Sent: Tuesday, August 22, 2017 3:16 AM
>> To: dev@mesos.apache.org
>> Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos
>> 
>> Hi Alexander,
>> 
>> in general, jemalloc and tcmalloc are very similar, and seem to be taking
>> ideas from each other (in fact the jeprof executable started as a copy of
>> pprof and there are still references the pprof documentation in some
>> comments)
>> 
>> From what I've seen, the main difference is that the profiling seems
>> better-suited to multi-threaded programs, in particular the profile file
>> format includes per-thread memory statistics and the profiling features can
>> be turned on and off individually per thread. From an API perspective, all
>> settings can be accessed by the mallctl() function, while it seems that
>> tcmalloc requires some options to be set by environment variable (
>> https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fgperftools.github.io%2Fgperftools%
>> 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com%
>> 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011
>> db47%7C1%7C0%7C636389937852256730&sdata=IQeb2%
>> 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I also
>> found the documentation to be more thorough.
>> 
>> But again, the two are very similar, so I think the main decision here
>> isn't whether to choose jemalloc or tcmalloc but whether to switch to a
>> custom memory allocator that has support for profiling heap memory usage.
>> 
>> 
>> On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas <alexander@mesosphere.io>
>> wrote:
>> 
>>> Hi Benno,
>>> 
>>> This does sound like a great addition to Mesos. Can you however
>>> explain how jemalloc is better than tcmalloc? I think that for such
>>> important change, we probably need some more information.
>>> 
>>> Your comment in MESOS-7876 mentions that we already have tcmalloc
>>> since it is part of gperftools, so I would like to have a whole
>>> picture of the advantages and disadvantages of both options.
>>> 
>>> Alexander Rojas
>>> alexander@mesosphere.io
>>> 
>>> 
>>> 
>>> 
>>>> On 18. Aug 2017, at 12:49, Benno Evers <bevers@mesosphere.com> wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> I would like to propose bundling jemalloc as a new dependency under
>>>> `3rdparty/`, and to link Mesos against this new memory allocator by
>>>> default.
>>>> 
>>>> 
>>>> # Motivation
>>>> 
>>>> The Mesos master and agent binaries are, ideally, very long-running
>>>> processes. This makes them susceptible to memory issues, because
>>>> even small leaks have a chance to build up over time to the point
>>>> where they become problematic.
>>>> 
>>>> We have seen several such issues on our internal Mesos
>>>> installations, for example
>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
>>>> ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7748&data=02%7C01%7CJeff.Coff
>>>> ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f
>>>> 141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=L016YGyEkK5
>>>> 0WtvhgSNS%2FT5ntkkd9qINorRI2Utp5lk%3D&reserved=0
>>>> or https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMESOS-
>> 7800&data=02%7C01%7CJeff.Coffler%40microsoft.com%
>> 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011
>> db47%7C1%7C0%7C636389937852266742&sdata=IrzDO6o1VL9a8eGJIW3jKbWXk6U4fH
>> Fn3Xbn4po1r3c%3D&reserved=0.
>>>> 
>>>> I imagine any organization running Mesos for an extended period of
>>>> time has had its share of similar issues, so I expect this proposal
>>>> to be useful for the whole community.
>>>> 
>>>> 
>>>> # Why jemalloc?
>>>> 
>>>> Given that memory issues tend to be most visible after a given
>>>> process has been running for a long time, it would be great to have
>>>> the option to enable heap tracking and profiling at runtime, without
>>>> having to restart the process. (This ability could then be connected
>>>> to a Mesos endpoint, similar to how we can adjust the log level at
>>>> runtime)
>>>> 
>>>> The two production-quality memory allocators that have this ability
>>>> currently seem to be tcmalloc and jemalloc. Of these, jemalloc does
>>>> produce in our experience better and more detailed statistics.
>>>> 
>>>> 
>>>> # What is the impact on users who do not need this feature?
>>>> 
>>>> Naturally, not every single user of Mesos will have a need for this
>>>> feature. To ensure these users would not experience serious
>>>> performance regressions as a result of this change, we conducted a
>>>> preliminary set of benchmarks whose results are collected under
>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss
>>>> ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7876&data=02%7C01%7CJeff.Coff
>>>> ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f
>>>> 141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=RsZcAGuFm%2
>>>> Bw2PPLgMql%2B9vVgkFQrZZFJYdPGcBODsCU%3D&reserved=0
>>>> 
>>>> It turns out that we could probably even expect a small speedup (1%
>>>> - 5%) as a nice side-effect of this change.
>>>> 
>>>> Users who compile Mesos themselves would of course have the option
>>>> to disable jemalloc at configuration time or replace it with their
>>>> memory allocator of choice.
>>>> 
>>>> 
>>>> 
>>>> I'm looking forward to hear any thoughts and comments.
>>>> 
>>>> 
>>>> Thanks,
>>>> --
>>>> Benno Evers
>>>> Software Engineer, Mesosphere
>>> 
>>> 
>> 
>> 
>> --
>> Benno Evers
>> Software Engineer, Mesosphere
>> 
> 
> 
> 
> -- 
> Benno Evers
> Software Engineer, Mesosphere


Mime
View raw message