Mailing-List: contact user-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@mesos.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAFt=ROP6ffdx1Cj9Fd6Sz+hscJgsx5+cmmcJackWLZBjdVJkEw@mail.gmail.com>
References: 
 <CAEoEa4Zch2Vz10XDBhy4WUsVqRjGa+nOLTxfzt=paLr-m-pM2A@mail.gmail.com>
 <CAEoEa4ahoFKmT4aiptZ2dKN5Obk9Y+hFJ=mh=L=PLQNhZrc4Mw@mail.gmail.com>
 <CAFt=ROP6ffdx1Cj9Fd6Sz+hscJgsx5+cmmcJackWLZBjdVJkEw@mail.gmail.com>
From: June Taylor <june@umn.edu>
Date: Mon, 25 Apr 2016 13:15:11 -0500
Message-ID: 
 <CAEoEa4a6tH5A5Gvh7NV6F70Ew2_7dDFnGtmfAtZnA1990hSXkg@mail.gmail.com>
Subject: Re: Setting ulimits on mesos-slave
To: user@mesos.apache.org
Content-Type: multipart/alternative; boundary=001a114318f00364ad0531532ae5

--001a114318f00364ad0531532ae5
Content-Type: text/plain; charset=UTF-8

What I'm saying is even putting them within the upstart script, per the
Mesos documentation, isn't working for the file block limit. We're still
getting 8MB useable, and as a result executors fail when attempting to
write larger files.


Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota

On Mon, Apr 25, 2016 at 11:53 AM, haosdent <haosdent@gmail.com> wrote:

> If you set in your upstart script, it isn't system wide and only effective
> in that session. I think need change /etc/security/limits.conf
> and /etc/sysctl.conf to make your ulimit work globally.
>
> On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <june@umn.edu> wrote:
>
>> Somewhere an 8MB maximum file size is being applied on just one of our
>> slaves, for example.
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Mon, Apr 25, 2016 at 11:42 AM, June Taylor <june@umn.edu> wrote:
>>
>>> We are operating a 6-node cluster running on Ubuntu, and have noticed
>>> that the ulimit settings within the slave context are difficult to set and
>>> predict.
>>>
>>> The documentation is a bit unclear on this point, as well.
>>>
>>> We have had some luck adding a configuration line to
>>> /etc/init/mesos-slave.conf as follows:
>>> limit nofile 20000 20000
>>> limit fsize unlimited unlimited
>>>
>>> The nofile limit seems to be respected, however the fsize limit does not.
>>>
>>> It is also mysterious that the system-wide limits are not inherited by
>>> the slave process. We would prefer to set all of these system-wide and have
>>> mesos-slave observe them.
>>>
>>> Can you please advise where you are setting your ulimits for the
>>> mesos-slave if it is working for you?
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>

--001a114318f00364ad0531532ae5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">What I&#39;m saying is even putting them within the upstar=
t script, per the Mesos documentation, isn&#39;t working for the file block=
 limit. We&#39;re still getting 8MB useable, and as a result executors fail=
 when attempting to write larger files.</div><div class=3D"gmail_extra"><br=
 clear=3D"all"><div><div class=3D"gmail_signature"><div dir=3D"ltr"><div><d=
iv dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><br></div><div d=
ir=3D"ltr">Thanks,</div><div dir=3D"ltr">June Taylor</div></div><div>System=
 Administrator, Minnesota Population Center</div><div>University of Minneso=
ta</div></div></div></div></div></div></div></div>
<br><div class=3D"gmail_quote">On Mon, Apr 25, 2016 at 11:53 AM, haosdent <=
span dir=3D"ltr">&lt;<a href=3D"mailto:haosdent@gmail.com" target=3D"_blank=
">haosdent@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_qu=
ote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex=
"><div dir=3D"ltr">If you set in your upstart script, it isn&#39;t system w=
ide and only=C2=A0effective in that session. I think need change=C2=A0/etc/=
security/limits.conf and=C2=A0/etc/sysctl.conf to make your ulimit work glo=
bally.</div><div class=3D"gmail_extra"><div><div class=3D"h5"><br><div clas=
s=3D"gmail_quote">On Tue, Apr 26, 2016 at 12:43 AM, June Taylor <span dir=
=3D"ltr">&lt;<a href=3D"mailto:june@umn.edu" target=3D"_blank">june@umn.edu=
</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin=
:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">S=
omewhere an 8MB maximum file size is being applied on just one of our slave=
s, for example.</div><div class=3D"gmail_extra"><span><br clear=3D"all"><di=
v><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><d=
iv dir=3D"ltr"><br></div><div dir=3D"ltr">Thanks,</div><div dir=3D"ltr">Jun=
e Taylor</div></div><div>System Administrator, Minnesota Population Center<=
/div><div>University of Minnesota</div></div></div></div></div></div></div>=
</div>
<br></span><div><div><div class=3D"gmail_quote">On Mon, Apr 25, 2016 at 11:=
42 AM, June Taylor <span dir=3D"ltr">&lt;<a href=3D"mailto:june@umn.edu" ta=
rget=3D"_blank">june@umn.edu</a>&gt;</span> wrote:<br><blockquote class=3D"=
gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-=
left:1ex"><div dir=3D"ltr">We are operating a 6-node cluster running on Ubu=
ntu, and have noticed that the ulimit settings within the slave context are=
 difficult to set and predict.<div><br></div><div>The documentation is a bi=
t unclear on this point, as well.</div><div><br></div><div>We have had some=
 luck adding a configuration line to /etc/init/mesos-slave.conf as follows:=
</div><div><div>limit nofile 20000 20000</div><div>limit fsize unlimited un=
limited</div><div><br></div><div>The nofile limit seems to be respected, ho=
wever the fsize limit does not.</div><div><br></div><div>It is also mysteri=
ous that the system-wide limits are not inherited by the slave process. We =
would prefer to set all of these system-wide and have mesos-slave observe t=
hem.</div><div><br></div><div>Can you please advise where you are setting y=
our ulimits for the mesos-slave if it is working for you?</div><div><div><d=
iv dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D=
"ltr"><br></div><div dir=3D"ltr">Thanks,</div><div dir=3D"ltr">June Taylor<=
/div></div><div>System Administrator, Minnesota Population Center</div><div=
>University of Minnesota</div></div></div></div></div></div></div></div>
</div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div></div></div><span c=
lass=3D"HOEnZb"><font color=3D"#888888">-- <br><div>Best Regards,<br><div>H=
aosdent Huang</div></div>
</font></span></div>
</blockquote></div><br></div>

--001a114318f00364ad0531532ae5--