mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avinash Sridharan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-6143) resolv.conf is not copied when using the Mesos containerizer with a Docker image
Date Thu, 13 Oct 2016 02:42:20 GMT

    [ https://issues.apache.org/jira/browse/MESOS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570610#comment-15570610
] 

Avinash Sridharan commented on MESOS-6143:
------------------------------------------

[~jpinkul] tried Mesos 1.0.0 with an alpine image and I don't see this issue. I ran the Mesos
1.0.0 on Debian 8 with the following configuration:

Mesos master:
```
sudo /usr/sbin/mesos-master --ip=172.31.12.173 --port=5050 --log_dir=/var/log/mesos --work_dir=/var/lib/mesos
```

Mesos agent:
```
sudo /usr/sbin/mesos-slave                               --ip=172.31.12.173              
                           --master=172.31.12.173:5050       --isolation=filesystem/linux,docker/runtime
            --work_dir=/var/lib/mesos                               --image_providers=docker
```

mesos-execute:
```
mesos-execute --master=172.31.12.173:5050 --name=dns-test --docker_image=alpine --command="sleep
10000"
```

Ran the `nsenter` command on the container and verified that the /etc/resolv.conf in the new
mnt namespace is the same as that on the hostfs:
```
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$
ps aux | grep mesos
admin      550  0.0  0.0  25540  2712 pts/0    S+   01:52   0:00 screen -S mesos
admin      551  0.0  0.0  26900  3856 ?        Ss   01:52   0:00 SCREEN -S mesos
root      1522  0.0  0.0  40540  3440 pts/1    S+   01:59   0:00 sudo /usr/sbin/mesos-master
--ip=172.31.12.173 --port=5050 --log_dir=/var/log/mesos --work_dir=/var/lib/mesos
root      1523  0.0  0.2 904584 33708 pts/1    Sl+  01:59   0:01 /usr/sbin/mesos-master --ip=172.31.12.173
--port=5050 --log_dir=/var/log/mesos --work_dir=/var/lib/mesos
root      1538  0.0  0.0  40540  3428 pts/2    S+   02:01   0:00 sudo /usr/sbin/mesos-slave
--ip=172.31.12.173 --master=172.31.12.173:5050 --isolation=filesystem/linux,docker/runtime
--work_dir=/var/lib/mesos --image_providers=docker
root      1539  0.1  0.2 818592 35980 pts/2    Sl+  02:01   0:03 /usr/sbin/mesos-slave --ip=172.31.12.173
--master=172.31.12.173:5050 --isolation=filesystem/linux,docker/runtime --work_dir=/var/lib/mesos
--image_providers=docker
admin     2045  0.0  0.1 817848 30744 pts/3    Sl+  02:21   0:00 mesos-execute --master=172.31.12.173:5050
--name=dns-test --docker_image=alpine --command=sleep 10000
root      2058  0.2  0.1 816488 30068 ?        Ssl  02:21   0:01 mesos-executor --launcher_dir=/usr/libexec/mesos
--sandbox_directory=/mnt/mesos/sandbox --user=admin --rootfs=/var/lib/mesos/provisioner/containers/dcbe7b8a-e430-4b7f-98eb-d7f62c0c0f87/backends/copy/rootfses/0d8eceac-721b-4a3c-a68e-34ecc30cd718
admin     2101  0.0  0.0  12728  2168 pts/4    S+   02:31   0:00 grep mesos
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$
sudo nsenter -t 2058 -m cat /etc/alpine-release
3.4.3
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$
ls /etc/
adduser.conf            cloud           deluser.conf  grub.d       initramfs-tools  ld.so.cache
    lvm             mke2fs.conf     opt         python2.7  rcS.d         sgml            
          subuid-      udev
adjtime                 cron.d          dhcp          gshadow      inputrc          ld.so.conf
     machine-id      modprobe.d      os-release  python3    resolv.conf   shadow         
           sudoers      ufw
alternatives            cron.daily      dkms          gshadow-     insserv          ld.so.conf.d
   magic           modules         pam.conf    python3.4  rmt           shadow-          
         sudoers.d    vim
apt                     cron.hourly     dpkg          gss          insserv.conf     libaudit.conf
  magic.mime      modules-load.d  pam.d       rc0.d      rpc           shells            
        sysconfig    wgetrc
bash.bashrc             cron.monthly    emacs         host.conf    insserv.conf.d   locale.alias
   mailcap         motd            passwd      rc1.d      rsyslog.conf  skel             
         sysctl.conf  xdg
bash_completion.d       crontab         environment   hostname     iproute2         locale.gen
     mailcap.order   mtab            passwd-     rc2.d      rsyslog.d     ssh            
           sysctl.d     xml
bindresvport.blacklist  cron.weekly     fstab         hosts        issue            localtime
      manpath.config  nanorc          perl        rc3.d      screenrc      ssl           
            systemd
binfmt.d                dbus-1          gai.conf      hosts.allow  issue.net        logcheck
       mesos           network         profile     rc4.d      securetty     staff-group-for-usr-local
 terminfo
ca-certificates         debconf.conf    groff         hosts.deny   java             login.defs
     mesos-master    networks        profile.d   rc5.d      security      subgid         
           timezone
ca-certificates.conf    debian_version  group         init         kernel           logrotate.conf
 mesos-slave     newt            protocols   rc6.d      selinux       subgid-            
       tmpfiles.d
calendar                default         group-        init.d       ldap             logrotate.d
    mime.types      nsswitch.conf   python      rc.local   services      subuid          
          ucf.conf
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$
ls /etc/apline-release
ls: cannot access /etc/apline-release: No such file or directory
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$
sudo nsenter -t 2058 -m cat /etc/resolv.conf
domain us-west-2.compute.internal
search us-west-2.compute.internal
nameserver 172.31.0.2
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$
sudo cat /etc/resolv.conf
domain us-west-2.compute.internal
search us-west-2.compute.internal
nameserver 172.31.0.2
admin@ip-172-31-12-173:/var/lib/mesos/slaves/70a7875b-aecc-43f0-8aea-2a239d4e97da-S0/frameworks$
```
As you can see in the above output for PID 2058 , just to verify that we are looking at two
different mount spaces listed the /etc of the hostfs and the /etc in the mnt namespace seen
by PID 2058, and you can see the `alpine-release` file under /etc for PID 2058.

A few more questions on your test, given that you are saying the `resolv.conf` in your setup
is empty in the container how is the `ping` that you are running even succeeding. If `resolv.conf`
hadn't been setup properly, the container should have returned a TASK_FAILED?


> resolv.conf is not copied when using the Mesos containerizer with a Docker image
> --------------------------------------------------------------------------------
>
>                 Key: MESOS-6143
>                 URL: https://issues.apache.org/jira/browse/MESOS-6143
>             Project: Mesos
>          Issue Type: Bug
>          Components: containerization, isolation
>    Affects Versions: 1.0.0
>         Environment: OS: Debian Jessie
> Mesos version: 1.0.0
>            Reporter: Justin Pinkul
>            Assignee: Avinash Sridharan
>
> When using the Mesos containierizer, host networking and a Docker image {{resolv.conf}}
is not copied from the host. The only piece of Mesos code that copies these file is currently
in the {{network/cni}} isolator so I tried turning this on, by setting {{isolation=network/cni,namespaces/pid,docker/runtime,cgroups/devices,gpu/nvidia,cgroups/cpu,disk/du,filesystem/linux}},
but the issue still remained. I suspect this might be related to not setting {{network_cni_config_dir}}
and {{network_cni_plugins_dir}} but it seems incorrect that these flags would be required
to use host networking.
> Here is how I am able to reproduce this issue:
> {code}
> mesos-execute --master=mesosmaster1:5050 \
> 		--name=dns-test \
> 		--docker_image=my-docker-image:1.1.3 \
> 		--command="bash -c 'ping google.com; while ((1)); do date; sleep 10; done'"
> # Find the PID of mesos-executor's child process and enter it
> nsenter -m -u -i -n -p -r -w -t $PID
> # This file will be empty
> cat /etc/resolv.conf
> {code}
> {code:title=Mesos agent log}
> I0908 17:39:24.599149 181564 slave.cpp:1688] Launching task dns-test for framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
> I0908 17:39:24.599567 181564 paths.cpp:528] Trying to chown '/mnt/01/mesos_work/slaves/67025326-9dfd-4cbb-a008-454a40bce2f5-S2/frameworks/51831498-0902-4ae9-a1ff-4396f8b8d823-0006/executors/dns-test/runs/52bdce71-04b0-4440-bb71-cb826f0635c6'
to user 'root'
> I0908 17:39:24.603970 181564 slave.cpp:5748] Launching executor dns-test of framework
51831498-0902-4ae9-a1ff-4396f8b8d823-0006 with resources cpus(*):0.1; mem(*):32 in work directory
'/mnt/01/mesos_work/slaves/67025326-9dfd-4cbb-a008-454a40bce2f5-S2/frameworks/51831498-0902-4ae9-a1ff-4396f8b8d823-0006/executors/dns-test/runs/52bdce71-04b0-4440-bb71-cb826f0635c6'
> I0908 17:39:24.604178 181564 slave.cpp:1914] Queuing task 'dns-test' for executor 'dns-test'
of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
> I0908 17:39:24.604284 181571 docker.cpp:1020] Skipping non-docker container
> I0908 17:39:24.604532 181578 containerizer.cpp:781] Starting container '52bdce71-04b0-4440-bb71-cb826f0635c6'
for executor 'dns-test' of framework '51831498-0902-4ae9-a1ff-4396f8b8d823-0006'
> I0908 17:39:24.606972 181571 provisioner.cpp:294] Provisioning image rootfs '/mnt/01/mesos_work/provisioner/containers/52bdce71-04b0-4440-bb71-cb826f0635c6/backends/copy/rootfses/db97ba50-c9f0-45e7-8a39-871e4038abf9'
for container 52bdce71-04b0-4440-bb71-cb826f0635c6
> I0908 17:39:30.037472 181564 cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus 0.1)
for container 52bdce71-04b0-4440-bb71-cb826f0635c6
> I0908 17:39:30.038415 181560 linux_launcher.cpp:281] Cloning child process with flags
= CLONE_NEWNS | CLONE_NEWPID
> I0908 17:39:30.040742 181560 systemd.cpp:96] Assigned child process '190563' to 'mesos_executors.slice'
> I0908 17:39:30.161613 181576 slave.cpp:2902] Got registration for executor 'dns-test'
of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006 from executor(1)@10.191.4.65:43707
> I0908 17:39:30.162148 181563 disk.cpp:171] Updating the disk resources for container
52bdce71-04b0-4440-bb71-cb826f0635c6 to cpus(*):0.1; mem(*):32; gpus(*):2
> I0908 17:39:30.162648 181566 cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus 0.1)
for container 52bdce71-04b0-4440-bb71-cb826f0635c6
> I0908 17:39:30.162822 181574 slave.cpp:2079] Sending queued task 'dns-test' to executor
'dns-test' of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006 at executor(1)@10.191.4.65:43707
> I0908 17:39:30.168383 181570 slave.cpp:3285] Handling status update TASK_RUNNING (UUID:
319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
from executor(1)@10.191.4.65:43707
> I0908 17:39:30.169019 181577 status_update_manager.cpp:320] Received status update TASK_RUNNING
(UUID: 319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
> I0908 17:39:30.169173 181576 slave.cpp:3678] Forwarding the update TASK_RUNNING (UUID:
319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
to master@10.191.248.194:5050
> I0908 17:39:30.169242 181576 slave.cpp:3588] Sending acknowledgement for status update
TASK_RUNNING (UUID: 319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
to executor(1)@10.191.4.65:43707
> I0908 17:39:30.171314 181565 status_update_manager.cpp:392] Received status update acknowledgement
(UUID: 319e0235-01b9-42ce-a2f8-ed9fc33de150) for task dns-test of framework 51831498-0902-4ae9-a1ff-4396f8b8d823-0006
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message