Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E00D11451 for ; Fri, 28 Mar 2014 18:06:17 +0000 (UTC) Received: (qmail 79837 invoked by uid 500); 28 Mar 2014 18:06:16 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 79655 invoked by uid 500); 28 Mar 2014 18:06:15 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 79641 invoked by uid 99); 28 Mar 2014 18:06:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Mar 2014 18:06:14 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ahl@box.com designates 209.85.212.176 as permitted sender) Received: from [209.85.212.176] (HELO mail-wi0-f176.google.com) (209.85.212.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Mar 2014 18:06:08 +0000 Received: by mail-wi0-f176.google.com with SMTP id r20so1031937wiv.9 for ; Fri, 28 Mar 2014 11:05:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=4ildC6fmh9ys55I/nkUAogVhbn9WGzpno1TVW5p2KH0=; b=Z+durX3gf6wmWB0L+wEz8Xc4Ze2ScTs07YtJqvSGr60Fe7nx7RHsXyd+TiGXihKgBL 4w3JWNJACHAHLRvVdCwV8eMTW34Z/wYSOkX7LGY4vCu8vNuG/g/u3kqKNy9O6KBKlppT y0Tt8Kgybq8XFpBr2H6EvRnr+lnHXCv7E9yFE8w9OTZS4HrVtZkEKjfgPc9bh0IFzOra /r3VSHWd2ljIVHnYvXVaRCTCFHnxGF64pqxiCMhE8uqFR+BT0M7jS9ohnJWsd21K3u/8 WTJQV+aRtnpJ/2+uRkjw1hBulxF46lM7iLYDnQQ9UkQnYiVTW/5zxZxzhYNQLxk5wXzG nDwA== X-Gm-Message-State: ALoCoQm7Gc4/wAHCimy2FBnUEMSei+kcOzcz25Ghj1FALmqkdtxvhBz9bn4LBxjUBubrPNEbQuPG X-Received: by 10.194.188.68 with SMTP id fy4mr133438wjc.30.1396029947198; Fri, 28 Mar 2014 11:05:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.118.10 with HTTP; Fri, 28 Mar 2014 11:05:27 -0700 (PDT) From: Tony Landells Date: Fri, 28 Mar 2014 11:05:27 -0700 Message-ID: Subject: help with cgroups isolator on Scientific Linux To: user@mesos.apache.org Content-Type: multipart/alternative; boundary=047d7bb03f9a1951f304f5ae8de2 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bb03f9a1951f304f5ae8de2 Content-Type: text/plain; charset=ISO-8859-1 Hi, I'm trying to get Mesos up as a PoC at work to see if it's something we could use effectively. I'm running SL 6.4, installed libcgroup and ran cgconfig which creates the cgroup mounts under /cgroup: [tlandells@tlandells-mesos02.dev ~]$ cat /proc/mounts | grep cgroup > cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0 > cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0 > cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 > cgroup /cgroup/memory cgroup rw,relatime,memory 0 0 > cgroup /cgroup/devices cgroup rw,relatime,devices 0 0 > cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0 > cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0 > cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0 I'm using the mesos_0.17.0_x86_64.rpm package downloaded from mesosphere.ioand I can't start a slave fine if I use process isolation, but if I try cgroups I get the following in /var/log/messages: 2014-03-28T10:46:05.194958-07:00 tlandells-mesos02 mesos-slave[6242]: I0328 > 10:46:05.194540 6242 main.cpp:118] Build: 2014-02-27 15:07:00 by root > 2014-03-28T10:46:05.194976-07:00 tlandells-mesos02 mesos-slave[6242]: > I0328 10:46:05.194810 6242 main.cpp:120] Creating "cgroups" isolator > 2014-03-28T10:46:05.195267-07:00 tlandells-mesos02 mesos-slave[6242]: > I0328 10:46:05.195050 6242 main.cpp:132] Starting Mesos slave > 2014-03-28T10:46:05.195657-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@712: Client > environment:zookeeper.version=zookeeper C client 3.4.5 > 2014-03-28T10:46:05.195667-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@716: Client > environment:host.name=tlandells-mesos02.dev.box.net. > 2014-03-28T10:46:05.195671-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@723: Client > environment:os.name=Linux > 2014-03-28T10:46:05.195674-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@724: Client > environment:os.arch=2.6.32-358.18.1.el6.x86_64 > 2014-03-28T10:46:05.195678-07:00 tlandells-mesos02 mesos-slave[6242]: > I0328 10:46:05.195552 6242 slave.cpp:111] Slave started on 1)@ > 10.3.18.184:5051 > 2014-03-28T10:46:05.195681-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@725: Client > environment:os.version=#1 SMP Tue Aug 27 14:23:09 CDT 2013 > 2014-03-28T10:46:05.195773-07:00 tlandells-mesos02 mesos-slave[6242]: > I0328 10:46:05.195708 6242 slave.cpp:211] Slave resources: cpus(*):2; > mem(*):2808; disk(*):43161; ports(*):[31000-32000] > 2014-03-28T10:46:05.196105-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@733: Client > environment:user.name=tlandells > 2014-03-28T10:46:05.196115-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@741: Client > environment:user.home=/root > 2014-03-28T10:46:05.196119-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@753: Client > environment:user.dir=/home/tlandells > 2014-03-28T10:46:05.196122-07:00 tlandells-mesos02 mesos-slave[6242]: > 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@zookeeper_init@786: > Initiating client connection, host=tlandells-mesos01.dev.box.net:2181, > tlandells-mesos02.dev.box.net:2181,tlandells-mesos03.dev.box.net:2181sessionTimeout=10000 watcher=0x7fc602fc8cb0 sessionId=0 > sessionPasswd= context=0x7fc5d8003b90 flags=0 > 2014-03-28T10:46:05.197040-07:00 tlandells-mesos02 mesos-slave[6242]: > I0328 10:46:05.196420 6269 cgroups_isolator.cpp:223] Using /cgroup as > cgroups hierarchy root > 2014-03-28T10:46:05.197058-07:00 tlandells-mesos02 mesos-slave[6242]: > Failed to mount cgroups hierarchy at '/cgroup' because we could not remove > existing directory: Directory not empty The command being invoked by /usr/bin/mesos-init-wrapper is: > /usr/local/sbin/mesos-slave --master=zk:// > tlandells-mesos01.dev.box.net:2181,tlandells-mesos02.dev.box.net:2181, > tlandells-mesos03.dev.box.net:2181/mesos --log_dir=/var/log/mesos > --isolation=cgroups I haven't worked with cgroups before so I'm not clear if it's okay to have Mesos using the default mounts or whether I should have something special setup. I mean it's clear that Mesos wouldn't be able to empty out /cgroup because it contains all the cgroup mounts... I saw a similar thread in the archives from about 6 months ago where the issue was apparently a clash between Mesos and Docker, but I'm only trying to run Mesos. The only thing that might be funky is that these are VMware instances I'm currently using for testing. Does anyone have any experience running the cgroups isolator in SL (or even RH, since it's basically just a rebadge) who can tell me what I'm doing wrong? Thanks, Tony --047d7bb03f9a1951f304f5ae8de2 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

I'm trying to get Mesos up as a PoC at work= to see if it's something we could use effectively.

I'm runn= ing SL 6.4, installed libcgroup and ran cgconfig which creates the cgroup m= ounts under /cgroup:

[tlandells@tlandells-mesos02.dev ~]$ cat /proc/mounts = | grep cgroup
cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /cgroup/cpu c= group rw,relatime,cpu 0 0
cgroup /cgroup/cpuacct cgroup rw,relatime,cpua= cct 0 0
cgroup /cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /c= group/devices cgroup rw,relatime,devices 0 0
cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /cgroup/net= _cls cgroup rw,relatime,net_cls 0 0
cgroup /cgroup/blkio cgroup rw,relat= ime,blkio 0 0

I'm using the mesos_0.17.= 0_x86_64.rpm package downloaded from mesos= phere.io and I can't start a slave fine if I use process isolation,= but if I try cgroups I get the following in /var/log/messages:

2014-03-28T10:46:05.194958-07:00 tland= ells-mesos02 mesos-slave[6242]: I0328 10:46:05.194540=A0 6242 main.cpp:118]= Build: 2014-02-27 15:07:00 by root
2014-03-28T10:46:05.194976-07:00 tlandells-mesos02 mesos-slave[6242]: I0328= 10:46:05.194810=A0 6242 main.cpp:120] Creating "cgroups" isolato= r
2014-03-28T10:46:05.195267-07:00 tlandells-mesos02 mesos-slave[6242]: = I0328 10:46:05.195050=A0 6242 main.cpp:132] Starting Mesos slave
2014-03-28T10:46:05.195657-07:00 tlandells-mesos02 mesos-slave[6242]: 2014-= 03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@712: Client enviro= nment:zookeeper.version=3Dzookeeper C client 3.4.5
2014-03-28T10:46:05.1= 95667-07:00 tlandells-mesos02 mesos-slave[6242]: 2014-03-28 10:46:05,195:62= 42(0x7fc5fafcb700):ZOO_INFO@log_env@716: Client environment:host.name=3Dtlandells-mesos02.dev.box.net.
2014-03-28T10:46:05.195671-07:00 tlandells-mesos02 mesos-slave[6242]: 2014-= 03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@723: Client enviro= nment:os.name=3DLinux
2014-03-28T10:46:05= .195674-07:00 tlandells-mesos02 mesos-slave[6242]: 2014-03-28 10:46:05,195:= 6242(0x7fc5fafcb700):ZOO_INFO@log_env@724: Client environment:os.arch=3D2.6= .32-358.18.1.el6.x86_64
2014-03-28T10:46:05.195678-07:00 tlandells-mesos02 mesos-slave[6242]: I0328= 10:46:05.195552=A0 6242 slave.cpp:111] Slave started on 1)@10.3.18.184:5051
2014-03-28T10:46:05.195681-07:= 00 tlandells-mesos02 mesos-slave[6242]: 2014-03-28 10:46:05,195:6242(0x7fc5= fafcb700):ZOO_INFO@log_env@725: Client environment:os.version=3D#1 SMP Tue = Aug 27 14:23:09 CDT 2013
2014-03-28T10:46:05.195773-07:00 tlandells-mesos02 mesos-slave[6242]: I0328= 10:46:05.195708=A0 6242 slave.cpp:211] Slave resources: cpus(*):2; mem(*):= 2808; disk(*):43161; ports(*):[31000-32000]
2014-03-28T10:46:05.196105-0= 7:00 tlandells-mesos02 mesos-slave[6242]: 2014-03-28 10:46:05,195:6242(0x7f= c5fafcb700):ZOO_INFO@log_env@733: Client environment:user.name=3Dtlandells
2014-03-28T10:46:05.196115-07:00 tlandells-mesos02 mesos-slave[6242]: 2014-= 03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@741: Client enviro= nment:user.home=3D/root
2014-03-28T10:46:05.196119-07:00 tlandells-mesos= 02 mesos-slave[6242]: 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO= @log_env@753: Client environment:user.dir=3D/home/tlandells
2014-03-28T10:46:05.196122-07:00 tlandells-mesos02 mesos-slave[6242]: 2014-= 03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@zookeeper_init@786: Initia= ting client connection, host=3Dtlandells-mesos01.dev.box.net:2181,tlandells-mesos02.dev.box.net:2181,tlandells-mesos03.dev.box.ne= t:2181 sessionTimeout=3D10000 watcher=3D0x7fc602fc8cb0 sessionId=3D0 se= ssionPasswd=3D<null> context=3D0x7fc5d8003b90 flags=3D0
2014-03-28T10:46:05.197040-07:00 tlandells-mesos02 mesos-slave[6242]: I0328= 10:46:05.196420=A0 6269 cgroups_isolator.cpp:223] Using /cgroup as cgroups= hierarchy root
2014-03-28T10:46:05.197058-07:00 tlandells-mesos02 mesos= -slave[6242]: Failed to mount cgroups hierarchy at '/cgroup' becaus= e we could not remove existing directory: Directory not empty

The command being invoked by /usr/bin/mesos-init-wrappe= r is:
/usr/local/sbin/mesos-slave --master=3Dzk://tlandells-mesos01.dev.box.net:2181<= /a>,tlandells-mesos02= .dev.box.net:2181,tlandells-mesos03.dev.box.net:2181/mesos --log_dir=3D/var/log/me= sos --isolation=3Dcgroups

I haven't worked with cgroups before so I'= ;m not clear if it's okay to have Mesos using the default mounts or whe= ther I should have something special setup. I mean it's clear that Meso= s wouldn't be able to empty out /cgroup because it contains all the cgr= oup mounts...

I saw a similar thread in the archives from about 6 mon= ths ago where the issue was apparently a clash between Mesos and Docker, bu= t I'm only trying to run Mesos.

The only thing= that might be funky is that these are VMware instances I'm currently u= sing for testing.

Does anyone have any experience running the cgroups iso= lator in SL (or even RH, since it's basically just a rebadge) who can t= ell me what I'm doing wrong?

Thanks,
Tony
--047d7bb03f9a1951f304f5ae8de2--