Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4D17C18BA8 for ; Fri, 18 Mar 2016 20:58:38 +0000 (UTC) Received: (qmail 91018 invoked by uid 500); 18 Mar 2016 20:58:37 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 90949 invoked by uid 500); 18 Mar 2016 20:58:37 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 90940 invoked by uid 99); 18 Mar 2016 20:58:37 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2016 20:58:37 +0000 Received: from mail-ob0-f178.google.com (mail-ob0-f178.google.com [209.85.214.178]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 7740D1A0046 for ; Fri, 18 Mar 2016 20:58:37 +0000 (UTC) Received: by mail-ob0-f178.google.com with SMTP id fz5so128674601obc.0 for ; Fri, 18 Mar 2016 13:58:37 -0700 (PDT) X-Gm-Message-State: AD7BkJIJEQm6QH83bJrO4NHHkpTvSnO2ERdMf1Tm9/adb2oUTVm3FtL/9J63uNI3UdUXutUxDW9PcIIG57jqyC1w X-Received: by 10.182.79.167 with SMTP id k7mr11779675obx.43.1458334716889; Fri, 18 Mar 2016 13:58:36 -0700 (PDT) MIME-Version: 1.0 Received: by 10.76.39.2 with HTTP; Fri, 18 Mar 2016 13:58:17 -0700 (PDT) In-Reply-To: References: From: Benjamin Mahler Date: Fri, 18 Mar 2016 13:58:17 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: How to kill tasks when memory exceeds the cgroup limit? To: user Content-Type: multipart/alternative; boundary=089e0139fc98c3af09052e590242 --089e0139fc98c3af09052e590242 Content-Type: text/plain; charset=UTF-8 Interesting, why does it take down the slaves? Because a lot of organizations run with swap disabled (e.g. for more deterministic performance), we originally did not set the swap limit at all. When we introduced the '--cgroups_limit_swap' flag we had to make it default to false initially in case any users were depending on the original behavior of no swap limit. Now that it's been available for some time, we can consider moving the default to true. This is actually reflected in the TODO alongside the flag: https://github.com/apache/mesos/blob/0.28.0/src/slave/flags.cpp#L331-L336 Want to send a patch? We'd need to communicate this change to the default behavior in the CHANGELOG and specify how users can keep the original behavior. Also, there's more we would need to do in the long term for use cases that desire swapping. The only support today is (1) no memory limits (2) memory limit and no swap limit (3) both memory and swap limits. You can imagine scenarios where users may want to control how much they're allowed to swap, or maybe we want to swap for non-latency sensitive containers. However, it's more complicated (the user and operator have to co-operate more, there are more ways to run things, etc), and so the general advice is to disable swap to keep things simple and deterministic. On Fri, Mar 18, 2016 at 11:34 AM, Dick Davies wrote: > Great! > > I'm not really sure why mesos even allows RSS limiting without VMEM, > it takes down slaves like the Black Death > if you accidentally deploy a 'leaker'. I'm sure there's a use case I'm > not seeing :) > > On 18 March 2016 at 16:27, Shiyao Ma wrote: > > Thanks. The limit_swap works. > --089e0139fc98c3af09052e590242 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Interesting, why does it take down the slaves?

Because a lot of organizations run with swap disabled (e.g. for mor= e deterministic performance), we originally did not set the swap limit at a= ll. When we introduced the '--cgroups_limit_swap' flag we had to ma= ke it default to false initially in case any users were depending on the or= iginal behavior of no swap limit. Now that it's been available for some= time, we can consider moving the default to true. This is actually reflect= ed in the TODO alongside the flag:


Want to send a patch? We'd need to communicate this c= hange to the default behavior in the CHANGELOG and specify how users can ke= ep the original behavior.

Also, there's more w= e would need to do in the long term for use cases that desire swapping. The= only support today is (1) no memory limits (2) memory limit and no swap li= mit (3) both memory and swap limits. You can imagine scenarios where users = may want to control how much they're allowed to swap, or maybe we want = to swap for non-latency sensitive containers. However, it's more compli= cated (the user and operator have to co-operate more, there are more ways t= o run things, etc), and so the general advice is to disable swap to keep th= ings simple and deterministic.
=
On Fri, Mar 18, 2016 at 11:34 AM, Dick Davie= s <dick@hellooperator.net> wrote:
Great!

I'm not really sure why mesos even allows RSS limiting without VMEM, it takes down slaves like the Black Death
if you accidentally deploy a 'leaker'. I'm sure there's a u= se case I'm
not seeing :)

On 18 March 2016 at 16:27, Shiyao Ma <i@i= ntroo.me> wrote:
> Thanks. The limit_swap works.

--089e0139fc98c3af09052e590242--