Mailing-List: contact user-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mesos.apache.org
Received-SPF: pass (nike.apache.org: domain of barton.tomas@gmail.com
 designates 209.85.223.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <54254C16.5010206@blue-yonder.com>
References: <54254C16.5010206@blue-yonder.com>
From: Tomas Barton <barton.tomas@gmail.com>
Date: Fri, 26 Sep 2014 15:15:04 +0200
Message-ID: 
 <CAPV2KRgZyQ1LoiJNY9qrtVh7MDhPQrWmsW6GNokKiSsGjGdbKg@mail.gmail.com>
Subject: Re: Problems with OOM
To: user <user@mesos.apache.org>
Content-Type: multipart/alternative; boundary=089e01536c54bea69a0503f7b5b6

--089e01536c54bea69a0503f7b5b6
Content-Type: text/plain; charset=UTF-8

Just to make sure, all slaves are running with:

--isolation='cgroups/cpu,cgroups/mem'

Is there something suspicious in mesos slave logs?

On 26 September 2014 13:20, Stephan Erb <stephan.erb@blue-yonder.com> wrote:

>  Hi everyone,
>
> I am having issues with the cgroups isolation of Mesos. It seems like
> tasks are prevented from allocating more memory than their limit. However,
> they are never killed.
>
>    - My scheduled task allocates memory in a tight loop. According to
>    'ps', once its memory requirements are exceeded it is not killed, but ends
>    up in the state D ("uninterruptible sleep (usually IO)").
>    - The task is still considered running by Mesos.
>    - There is no indication of an OOM in dmesg.
>    - There is neither an OOM notice nor any other output related to the
>    task in the slave log.
>    - According to htop, the system load is increased with a significant
>    portion of CPU time spend within the kernel. Commonly the load is so high
>    that all zookeeper connections time out.
>
> I am running Aurora and Mesos 0.20.1 using the cgroups isolation on Debian
> 7 (kernel 3.2.60-1+deb7u3). .
>
> Sorry for the somewhat unspecific error description. Still, anyone an idea
> what might be wrong here?
>
> Thanks and Best Regards,
> Stephan
>

--089e01536c54bea69a0503f7b5b6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Just to make sure, all slaves are running with:<div><br></=
div><div>--isolation=3D&#39;cgroups/cpu,cgroups/mem&#39;<br></div><div><br>=
</div><div>Is there something suspicious in mesos slave logs?</div></div><d=
iv class=3D"gmail_extra"><br><div class=3D"gmail_quote">On 26 September 201=
4 13:20, Stephan Erb <span dir=3D"ltr">&lt;<a href=3D"mailto:stephan.erb@bl=
ue-yonder.com" target=3D"_blank">stephan.erb@blue-yonder.com</a>&gt;</span>=
 wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor=
der-left:1px #ccc solid;padding-left:1ex">
 =20

   =20
 =20
  <div bgcolor=3D"#FFFFFF" text=3D"#000000">
    Hi everyone,<br>
    <br>
    I am having issues with the cgroups isolation of Mesos. It seems
    like tasks are prevented from allocating more memory than their
    limit. However, they are never killed. <br>
    <ul>
      <li>My scheduled task allocates memory in a tight loop. According
        to &#39;ps&#39;, once its memory requirements are exceeded it is no=
t
        killed, but ends up in the state D (&quot;uninterruptible sleep
        (usually IO)&quot;). </li>
      <li>The task is still considered running by Mesos.</li>
      <li>There is no indication of an OOM in dmesg.</li>
      <li>There is neither an OOM notice nor any other output related to
        the task in the slave log.</li>
      <li>According to htop, the system load is increased with a
        significant portion of CPU time spend within the kernel.
        Commonly the load is so high that all zookeeper connections time
        out.<br>
      </li>
    </ul>
    I am running Aurora and Mesos 0.20.1 using the cgroups isolation on
    Debian 7 (kernel 3.2.60-1+deb7u3). .<br>
    <br>
    Sorry for the somewhat unspecific error description. Still, anyone
    an idea what might be wrong here?<br>
    <br>
    Thanks and Best Regards,<br>
    Stephan<br>
  </div>

</blockquote></div><br></div>

--089e01536c54bea69a0503f7b5b6--