mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joris Van Remoortere (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (MESOS-6264) Investigate the high memory usage of the default executor.
Date Wed, 05 Oct 2016 15:46:21 GMT

    [ https://issues.apache.org/jira/browse/MESOS-6264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549111#comment-15549111
] 

Joris Van Remoortere edited comment on MESOS-6264 at 10/5/16 3:45 PM:
----------------------------------------------------------------------

cc [~vinodkone][~jieyu]
The bulk of this comes from loading in {{libmesos.so}}
We do this because the autoconf build treats libmesos as a dynamic dependency.
Since we load libmesos dynamically, there is no chance for the linker to strip unused code.
This means that all of the code in libmesos regardless of use gets loaded into resident memory.
In contrast the cmake build generates a static library for {{libmesos.a}}. This is then used
to build the {{mesos-executor}} binary without a dynamic dependency on libmesos. The benefit
of this approach is that the linker is able to strip out all unused code. In an optimized
build this is {{~10MB}}.

Some approaches for the quick win are:
# Consider using the cmake build. This only needs to be modified slightly to strip symbols
from the final executor binary {{-s}}.
# Modiy the autoconf build to build a {{libmesos.a}} so that we can statically link it in
to the {{mesos-executor}} binary and allow the linker to strip unused code.

Regardless of the above approach, {{libmesos}} would still be by far the largest contributor
of the {{RSS}}. This is for 2 reasons:
# Much of our code is structured such that the linker can't determine if it is unused. We
would need to adjust our patterns such that the unused code analyzer can do a better job.
# Much of our code is {{inlined}} or written such that it can't be optimized. 2 examples are:
## https://github.com/apache/mesos/blob/9beb8eae6408249cdb3e2f16ba68b31a00d3452c/3rdparty/libprocess/include/process/mime.hpp#L35-L154
This code could be moved to a {{.cpp}} file and should be a {{static const std::unordered_map<string,
string>}} that we {{insert(begin(), end())}} into {{types}}. This would reduce the size
of libmesos by {{~20KB}}!
## https://github.com/apache/mesos/blob/9beb8eae6408249cdb3e2f16ba68b31a00d3452c/3rdparty/libprocess/include/process/http.hpp#L453-L517
This code and sibling {{struct Request}} have auto-generated {{inlined}} destructors. These
are very expensive. Just declaring and then defining in the {{.cpp}} the default destructor
can remove another {{~20KB}} each from libmesos. There are plenty of other opportunities like
this scattered through the codebase. It's work to find them and the returns are small for
each, but end up adding to much of the {{9MB}} left over.


was (Author: jvanremoortere):
cc [~vinodkone][~jieyu]
The bulk of this comes from loading in {{libmesos.so}}
We do this because the autoconf build treats libmesos as a dynamic dependency.
Since we load libmesos dynamically, there is no chance for the linker to strip unused code.
This means that all of the code in libmesos regardless of use gets loaded into resident memory.
In contrast the cmake build generates a static library for {{libmesos.a}}. This is then used
to build the {{mesos-executor}} binary without a dynamic dependency on libmesos. The benefit
of this approach is that the linker is able to strip out all unused code. In an optimized
build this is {{~10MB}}.

Some approaches for the quick win are:
# Consider using the cmake build. This only needs to be modified slightly to strip symbols
from the final executor binary {{-s}}.
# Modiy the autoconf build to build a {{libmesos.a}} so that we can statically link it in
to the {{mesos-executor}} binary and allow the linker to strip unused code.

Regardless of the above approach, {{libmesos}} would still be by far the largest contributor
of the {{RSS}}. This is for 2 reasons:
# Much of our code is structured such that the linker can't determine if it is unused. We
would need to adjust our patterns such that the unused code analyzer can do a better job.
# Much of our code is {{inlined}} or written such that it can't be optimized. 2 examples are:
## https://github.com/apache/mesos/blob/9beb8eae6408249cdb3e2f16ba68b31a00d3452c/3rdparty/libprocess/include/process/mime.hpp#L35-L154
This code could be moved to a {{.cpp}} file and should be a {{static const std::unordered_map<string,
string>}} that we {{insert(begin(), end())}} into {{types}}. This would reduce the size
of libmesos by {{~20KB}}!
## https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/http.hpp#L453-L517
This code and sibling {{struct Request}} have auto-generated {{inlined}} destructors. These
are very expensive. Just declaring and then defining in the {{.cpp}} the default destructor
can remove another {{~20KB}} each from libmesos. There are plenty of other opportunities like
this scattered through the codebase. It's work to find them and the returns are small for
each, but end up adding to much of the {{9MB}} left over.

> Investigate the high memory usage of the default executor.
> ----------------------------------------------------------
>
>                 Key: MESOS-6264
>                 URL: https://issues.apache.org/jira/browse/MESOS-6264
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Anand Mazumdar
>              Labels: mesosphere
>             Fix For: 1.1.0
>
>         Attachments: pmap_output_for_the_default_executor.txt
>
>
> It seems that a default executor with two sleep tasks is using ~32 mb on average and
can sometimes lead to it being killed for some tests like {{SlaveRecoveryTest/0.ROOT_CGROUPS_ReconnectDefaultExecutor}}
on our internal CI. Attached the {{pmap}} output for the default executor. Please note that
the command executor memory usage is also pretty high (~26 mb).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message