aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oskar Stenman <oskar.sten...@magine.com>
Subject Re: Problem getting docker container running
Date Fri, 13 Mar 2015 22:05:08 GMT
The majority of the resolution to this happened on irc, #aurora / freenode,
thanks to Steve :)
i'm including it here "for logging" / searchability.

We're running all of mesos, aurora, thermos-observer, zookeeper and such in
docker containers, mesos-slave in a container managing other containers on
the same layer of "containerization" puts special requirements on the
mesos-slave container.
simple schematic showing how things are layered:
(host -> docker
  docker -> mesos-slave
  docker -> mesos-task
  docker  -> ...
)

This requires that directories in the mesos-slave is on the same path in
both the slave and on the host since mesos-slave mounts those directories
from docker (host's) perspective into the mesos-task.

The mesos-slave also seems to have it's own executor which keeps track of
the aurora-executor's pid.
If mesos-slave can't find the slave pid it will assume the task was lost.
(this was the main issue i was banging my head against since this isn't
logged anywhere if it fails, slave just prints "task_lost" and executor
just exits..)
This happens if mesos-slave is running in a docker-container where it will
be in a isolated pid-space and therefore will not be able to find the pid
of the slave-container's executor.
By running the docker-container containing mesos-slave with "--pid=host"
the mesos-slave will run in the same pid-space as the host, and will
therefore be able to find aurora-executor's pid, and things will magically
work.. :>

For this (mesos-slave in docker-container, tasks in other containers and
such) mesos-slave, thermos_observer and all slave/mesos tasks needs to be
able to access the same directories.
/sys needs to be mounted into the mesos-slave /sys directory so it can
manage cgroups.
/var/run/thermos needs to be mounted into thermos_observer's
/var/run/thermos, this will also be mounted into all slave-tasks
/var/run/thermos directory, i'm assuming so all slave-tasks can report back
to thermos_observer that they're running or not..
/tmp/mesos (or the directory configured in mesos-slave where all tasks
sandboxes are stored) needs to be mounted from host's /tmp/mesos into the
mesos-slave's /tmp/mesos, mesos-slave will mount this from the host into
each mesos-task container.

Mesos-slave docker config:
/etc/init/mesos-slave.conf
#
# This properly starts and stops a docker container through upstart.
#

description     "run mesos-slave"

respawn

script
docker pull docker-repo.service.consul:5000/trusty/mesos
. /etc/default/cluster
mkdir -p /tmp/mesos

docker create -i -t --name=consul_$UPSTART_JOB --net=host \
-v /run/thermos:/run/thermos \
-v /tmp/mesos:/tmp/mesos \
-v /sys:/sys \
-v /usr/bin/docker:/usr/bin/docker \
-v /var/run/docker.sock:/var/run/docker.sock \
-e ULIMIT="-n 8192" \
-e MESOS_resources="ports:[13000-15999]" \
-e MESOS_FRAMEWORKS_HOME=/data/services/mesos/frameworks \
-e MESOS_launcher_dir=/usr/libexec/mesos \
-e MESOS_attributes="role:slave;rack:$RACK;host:`hostname`" \
--pid=host \ndocker-repo.service.consul:5000/trusty/mesos \
/usr/sbin/mesos-slave
--master=zk://zookeeper.service.consul:2181/mesos/$CLUSTER
--containerizers=docker,mesos --log_dir=/var/lib/mesos/log
--executor_registration_timeout=5mins

exec docker start -a consul_$UPSTART_JOB
end script

post-stop script
docker stop -t 0 consul_$UPSTART_JOB
docker rm consul_$UPSTART_JOB
sleep 5
end script


This doesn't seem to work properly for now. something is weird with
stdout/stderr, i'll investigate a bit further.
/etc/init/thermos-executor.conf:
#
# This properly starts and stops a docker container through upstart.
#

description     "run thermos-observer"

respawn

script
docker pull docker-repo.service.consul:5000/trusty/mesos
. /etc/default/cluster
docker create -i -t --name=$UPSTART_JOB --net=host \
-v /var/run/thermos:/var/run/thermos \
docker-repo.service.consul:5000/trusty/mesos
/usr/local/bin/thermos_observer.pex --log_dir=/data/log/thermos-observer
--log_simple
exec docker start -a $UPSTART_JOB
end script

post-stop script
docker stop -t 0 $UPSTART_JOB
docker rm $UPSTART_JOB
end script


On Fri, Mar 13, 2015 at 6:18 PM, Steve Niemitz <steve@tellapart.com> wrote:

> I just noticed this line in your previous email:
> "Everything's included in the image, that's the image i'm running
> mesos-master,
> aurora, and mesos-slave from."
>
> Are you saying you're running the slave inside a docker container?  If so,
> mesos does not support running docker containers from inside another docker
> container.
>
> On Fri, Mar 13, 2015 at 1:04 PM, Oskar Stenman <oskar.stenman@magine.com>
> wrote:
>
> > Thanks for the reply!
> >
> > The only files i find in the workdir is stderr, stdout and
> > thermos_executor.pex
> > Nothing is cleaning the tmp directory, everything (except /data) is
> running
> > on ramdisk on these machines, it's the absolute minimal ubuntu "minbase"
> > (debootstrapped) installation i've found possible and still generally
> > usable. (around 100MB with docker and some other tools i include)
> >
> >
> > root@s1:~# ls
> >
> >
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781
> > -l
> > total 29000
> > -rw-r--r-x 1 root root      519 Mar 13 13:39 stderr
> > -rw-r--r-x 1 root root        0 Mar 13 13:39 stdout
> > -rwxr-xr-x 1 root root 29690639 Mar 13 13:39 thermos_executor.pex
> >
> > STDERR:
> > root@s1:~# cat
> >
> >
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781/stderr
> > WARNING: Logging before InitGoogleLogging() is written to STDERR
> > I0313 13:39:45.608578   258 fetcher.cpp:76] Fetching URI
> > '/usr/local/bin/thermos_executor.pex'
> > I0313 13:39:45.609048   258 fetcher.cpp:179] Copying resource from
> > '/usr/local/bin/thermos_executor.pex' to
> >
> >
> '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
> >
> > /Oskar
> >
> > On Fri, Mar 13, 2015 at 5:54 PM, Bill Farner <wfarner@apache.org> wrote:
> >
> > > When the task launches, a sandbox directory is created, in the above
> > e-mail
> > > it
> > > was
> > >
> >
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781.
> > > Is there anything else in that directory?  Please post anything logs
> you
> > > find in there as they may provide useful clues.
> > >
> > > Possibly unrelated, your slave work directory is /tmp.  I have observed
> > > that some environments a temp cleaner process runs that automatically
> > > deletes things under here.  This may not fix the issue at hand here,
> but
> > i
> > > suggest you move this outside /tmp as it is critical state for the
> slave.
> > >
> > >
> > > -=Bill
> > >
> > > On Fri, Mar 13, 2015 at 9:24 AM, Oskar Stenman <
> oskar.stenman@magine.com
> > >
> > > wrote:
> > >
> > > > I wasn't subscribed to the mailing-list (I'm subscribed now though)
> so
> > > > i'm sorry if this reply ends up in the wrong place..
> > > >
> > > > >From the mesos slave log it looks like the executor is failing.
> Most
> > > > likely the issue is your image doesn't have the native libraries
> needed
> > > to
> > > > run it.  The next step would be to look in the sandbox for a failed
> run
> > > > (you can find the path in the slave logs) and look at the stderr log
> > for
> > > > errors.
> > > >
> > > > Everything's included in the image, that's the image i'm running
> > > > mesos-master, aurora, and mesos-slave from.
> > > >
> > > > Stdout was included in the last email:
> > > >
> > > > WARNING: Logging before InitGoogleLogging() is written to STDERR
> > > > I0313 13:39:45.608578   258 fetcher.cpp:76] Fetching URI
> > > > '/usr/local/bin/thermos_executor.pex'
> > > > I0313 13:39:45.609048   258 fetcher.cpp:179] Copying resource from
> > > > '/usr/local/bin/thermos_executor.pex' to
> > > >
> > > >
> > >
> >
> '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
> > > >
> > > > Also, when relaunching the container manually:
> > > >
> > > >
> > > >    1. root@s1:/var/log/upstart# docker start -i
> > > > mesos-cdf5e59f-c2be-47ba-b30a-2a690657e248
> > > >    2. twitter.common.app debug: Initializing: twitter.common.log
> > > > (Logging subsystem.)
> > > >    3. Writing log files to disk in /mnt/mesos/sandbox
> > > >    4. I0313 14:50:36.273181     5 exec.cpp:132] Version: 0.21.1
> > > >    5. I0313 14:50:36.278043    29 exec.cpp:379] Executor asked to
> > > shutdown
> > > >    6. Killed
> > > >
> > > >
> > > > It seems to be really tricky to troubleshoot from my point of view as
> > > > i don't have any output at all.
> > > > Is the executor giving up immediately? is it even trying to connect
> to
> > > > the slave?
> > > > Parameters to the executor missing? (like job-config or something)
> > > > Directory doesn't contain what it wants?
> > > > Network config wrong?
> > > >
> > > > /Oskar
> > > >
> > > >
> > > > On Fri, Mar 13, 2015 at 3:08 PM, Oskar Stenman <
> > oskar.stenman@magine.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > I'm investigating aurora + mesos + docker and i'm stuck.
> > > > >
> > > > > I can create the hello world docker-task in aurora, it gets
> assigned
> > a
> > > > > slave, the docker-container is launched but the executor
> immediately
> > > > > terminates and it ends up in "task lost" state.
> > > > >
> > > > > Can anyone make any sense of this or tell me how to troubleshoot
> > > further?
> > > > >
> > > > >
> > > > > --
> > > > > [image: MagineTV]
> > > > >
> > > > > *Oskar Stenman*
> > > > > Network Architect
> > > > >
> > > > > *Magine TV*
> > > > > oskar.stenman@magine.com  |   Mob: +46 70 565 21 52
> > > > > Regeringsgatan 25  | 111 53 Stockholm, Sweden  |   www.magine.com
> > > > > <http://www.magine.com/>
> > > > >
> > > > > Privileged and/or Confidential Information may be contained in this
> > > > > message. If you are not the addressee indicated in this message
> > > > > (or responsible for delivery of the message to such a person), you
> > may
> > > > not
> > > > > copy or deliver this message to anyone. In such case,
> > > > > you should destroy this message and kindly notify the sender by
> reply
> > > > > email.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > [image: MagineTV]
> > > >
> > > > *Oskar Stenman*
> > > > Network Architect
> > > >
> > > > *Magine TV*
> > > > oskar.stenman@magine.com  |   Mob: +46 70 565 21 52
> > > > Regeringsgatan 25  | 111 53 Stockholm, Sweden  |   www.magine.com
> > > > <http://www.magine.com/>
> > > >
> > > > Privileged and/or Confidential Information may be contained in this
> > > > message. If you are not the addressee indicated in this message
> > > > (or responsible for delivery of the message to such a person), you
> may
> > > not
> > > > copy or deliver this message to anyone. In such case,
> > > > you should destroy this message and kindly notify the sender by reply
> > > > email.
> > > >
> > >
> >
> >
> >
> > --
> > [image: MagineTV]
> >
> > *Oskar Stenman*
> > Network Architect
> >
> > *Magine TV*
> > oskar.stenman@magine.com  |   Mob: +46 70 565 21 52
> > Regeringsgatan 25  | 111 53 Stockholm, Sweden  |   www.magine.com
> > <http://www.magine.com/>
> >
> > Privileged and/or Confidential Information may be contained in this
> > message. If you are not the addressee indicated in this message
> > (or responsible for delivery of the message to such a person), you may
> not
> > copy or deliver this message to anyone. In such case,
> > you should destroy this message and kindly notify the sender by reply
> > email.
> >
>



-- 
[image: MagineTV]

*Oskar Stenman*
Network Architect

*Magine TV*
oskar.stenman@magine.com  |   Mob: +46 70 565 21 52
Regeringsgatan 25  | 111 53 Stockholm, Sweden  |   www.magine.com
<http://www.magine.com/>

Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply email.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message