mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Sallou <olivier.sal...@irisa.fr>
Subject Re: volume / mount point error with Unified Containerizer
Date Mon, 23 May 2016 07:35:05 GMT


On 05/23/2016 09:33 AM, Olivier Sallou wrote:
>
> On 05/20/2016 03:26 PM, Guangya Liu wrote:
>> Since you are using docker image which means that your container will have
>> rootfs, so it is not required to have the absolute path exist, the linux
>> file system isolator will help create the path automatically
>> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
>>
>> Can you please share your framework? How did you set the volume part in
>> your framework?
> @Guangya
>
> I use Python API.
>
> Here is related code:
>
>             ....
>          # Define container volumes
>          for v in job['container']['volumes']:
>             volume = container.volumes.add()
>             volume.container_path = v['mount']
>             volume.host_path = v['path']
>             if v['acl'] == 'rw':
>                 volume.mode = 1 # mesos_pb2.Volume.Mode.RW
>             else:
>                 volume.mode = 2 # mesos_pb2.Volume.Mode.RO
>
>     => In my test case, I add 2 volumes from a host shared directory,
> mounted in container as /mnt/go-docker and /mnt/god-data.
>
>             ...
>             # Define docker  image and network
>             docker = mesos_pb2.ContainerInfo.MesosInfo()
>             docker.image.type = 2 # Docker
>             docker.image.docker.name ='centos:latest'
>             # Request an IP from a network module
>             network_info = container.network_infos.add()
>             network_info_name = 'sampletest'
>             # Get an IP V4 address
>             ip_address = network_info.ip_addresses.add()
>             ip_address.protocol = 1
>             # The network group to join
>             group = network_info.groups.append(network_info_name)
>             port_list = [22]
>             if port_list:
>                 for port in port_list:
>                     job['container']['port_mapping'].append({'host':
> port, 'container': port})
>             container.mesos.MergeFrom(docker)
>
> It results in error message:
>
> + mount -n --rbind
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> + mount -n --rbind
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> mount: mount point
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> does not exist
> Failed to execute a preparation shell command
>
>
> We can see the  .rootfs, but mnt/god-data under .rootfs fails. Local
> directory exists, it does not pre-exists in container. What is strange
> is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is present.
>
> Or, is the error message (.rootfs/mnt/god-data does not exist) simply a
> warning, and it creates it, and final error (Failed to execute a
> preparation shell command) not related (and unclear...)
Additional info: command to execute in container is located in one of
mounted volume.
>
> Olivier
>> Thanks,
>>
>> Guangya
>>
>> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <olivier.sallou@irisa.fr>
>> wrote:
>>
>>> ----- Mail original -----
>>>> De: "Gilbert Song" <gilbert@mesosphere.io>
>>>> À: "dev" <dev@mesos.apache.org>
>>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
>>>> Objet: Re: volume / mount point error with Unified Containerizer
>>>>
>>>> @Olivier,
>>>> In mesos 0.28.1, you are supposed to be able bind mount a volume from
>>>> the host into the mesos container. Did you specify a docker image (we
>>>> determine
>>>> the mount point differently depending whether the container has a
>>> rootfs)?
>>>
>>> Yes I specified an image, a Docker image URI.
>>>
>>>> How
>>>> do you specify your 'container_path' (the mount point in the container)?
>>> If
>>>> it is an
>>>> absolute path, we require that dir to be pre-existed. If it is a relative
>>>> path, we will
>>>> mkdir for it.
>>> It is an absolute path, but it does not exists in image (this is the
>>> issue). Images are custom Docker images (images containing tools for batch
>>> computing), and I want, for example, to mount some shared resources (user
>>> home dir, common data, etc.) in the image. Of course those directories do
>>> not pre-exists in container images as they are specific to the environment.
>>> Requiring existence of the directory in the image is not issue as it
>>> prevents using any existing image from a repo.
>>>
>>> When using Docker containerizer it works fine, I can mount any external
>>> storage in the container.
>>>
>>> Olivie
>>>
>>>
>>>> @Joshua,
>>>> Thank for posting your workaround on mesos. As I mentioned above, in
>>> 0.28.1
>>>> or
>>>> older, we only mkdir for container_path which is relative path (not
>>>> starting with "/").
>>>> Because if no rootfs specified for a mesos container, the container
>>> shares
>>>> the host
>>>> root filesystem. Obviously we don't want any random files to be created
>>>> implicitly
>>>> on your host fs.
>>>> From mesos 0.29 (release by the end of this month), we will mkdir the
>>> mount
>>>> point in the container except for the command task case that specify an
>>>> absolute
>>>> container_path without a rootfs. Because we simplify the mounting logic,
>>> and
>>>> sandbox bind mount will only be done in container mount namespace
>>> instead of
>>>> host mount namespace (what we did before). Please keep tuned.
>>>>
>>>> Cheers,
>>>> Gilbert
>>>>
>>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jcohen@apache.org> wrote:
>>>>
>>>>> Hi Olivier,
>>>>>
>>>>> I touched on this issue as part of
>>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be nice if
>>>>> Mesos
>>>>> automatically created container mount points if they don't already
>>> exist.
>>>>> In the meantime, as a workaround for this, I've updated my filesystem
>>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir -p
>>>>> /some/mount/point`). Not the best solution, but the only thing I've
>>> seen
>>>>> that works at the moment.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Joshua
>>>>>
>>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <gyliu513@gmail.com>
>>> wrote:
>>>>>> It's pretty simple for you from scratch with source code
>>>>>>
>>>>>>
>>> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
>>>>>> ;-)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Guangya
>>>>>>
>>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
>>> olivier.sallou@irisa.fr
>>>>>> wrote:
>>>>>>
>>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
>>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
>>> driver"
>>>>>>> code
>>>>>>>> was not in 0.28.1, can you please have a try with mesos master
>>> branch
>>>>>> if
>>>>>>>> you are only doing some test?
>>>>>>> this is indeed test only for the moment. But I will have to
>>>>>>> recompile/install mesos  :-(  (I used packages for install).
>>>>>>>
>>>>>>> I will try when possible, but thanks for the hint.
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Guangya
>>>>>>>>
>>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <gyliu513@gmail.com
>>>>>> wrote:
>>>>>>>>> Hi Olivier,
>>>>>>>>>
>>>>>>>>> I think that you need to enable "docker volume isolator"
if you
>>> want
>>>>>> use
>>>>>>>>> external storage with unified container I was writing
a document
>>>>> here
>>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can
have a try
>>>>>>> according
>>>>>>>>> to the document and post some comments there if you find
any
>>> issues.
>>>>>>>>> Also you can patch mesos-execute here
>>>>>>> https://reviews.apache.org/r/46762/ to
>>>>>>>>> have a try with mesos-execute.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Guangya
>>>>>>>>>
>>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
>>>>>>> olivier.sallou@irisa.fr>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Answering (partially) to myself.
>>>>>>>>>>
>>>>>>>>>> I seems issue is container_path does not exists inside
>>> container.
>>>>> On
>>>>>>>>>> Docker, path is created and mounted. With pure mesos,
>>>>> container_path
>>>>>>>>>> must exists.
>>>>>>>>>>
>>>>>>>>>> mesos.proto says: "If the path is an absolute path,
that path
>>> must
>>>>>>>>>> already exist."
>>>>>>>>>>
>>>>>>>>>> This is an issue however, using Docker images, the
path I want
>>> to
>>>>>> mount
>>>>>>>>>> does not exists, and it cannot be modified "on the
fly".
>>>>>>>>>>
>>>>>>>>>> Is there a workaround for this ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> I am trying unified containerizer on a single
server
>>>>> (master/slave)
>>>>>> on
>>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer
to
>>> mesos+docker
>>>>>>> image
>>>>>>>>>>> container.
>>>>>>>>>>>
>>>>>>>>>>> I have setup slave config as suggested in documentation:
>>>>>>>>>>>
>>>>>>>>>>> containerizers=docker,mesos
>>>>>>>>>>> image_providers=docker \
>>>>>>>>>>> isolation=filesystem/linux,docker/runtime
>>>>>>>>>>>
>>>>>>>>>>> However, when I execute my task with a volume
I have an error:
>>>>>>>>>>>
>>>>>>>>>>> ....
>>>>>>>>>>> + mount -n --rbind
>>>>>>>>>>>
>>> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
>>>>>>>>>>> + mount -n --rbind
>>>>>>>>>>>
>>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>>>>>>>>> mount: mount point
>>>>>>>>>>>
>>> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
>>>>>>>>>>> does not exist
>>>>>>>>>>> Failed to execute a preparation shell command
>>>>>>>>>>>
>>>>>>>>>>> Then, my task switches to FAILED.
>>>>>>>>>>>
>>>>>>>>>>> I define a local volume to bind mount in my "container"
>>>>>>>>>>>
>>> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
>>>>>>>>>>> => /mnt/god-data
>>>>>>>>>>> My directory exists on local server.
>>>>>>>>>>> In mesos UI, I can see the .rootfs directory
along stdout and
>>>>> stderr
>>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data
(empty).
>>>>>>>>>>>
>>>>>>>>>>> Running the same using Docker containerizer instead
of mesos
>>>>>>>>>>> containerizer (with a Docker image) works fine.
>>>>>>>>>>>
>>>>>>>>>>> It seems it fails to mount my local directory
in the
>>> container.
>>>>> Any
>>>>>>> idea
>>>>>>>>>>> of what is going wrong or how to debug this?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Olivier Sallou
>>>>>>>>>> IRISA / University of Rennes 1
>>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>>>>>>> Tel: 02.99.84.71.95
>>>>>>>>>>
>>>>>>>>>> gpg key id: 4096R/326D8438  (keyring.debian.org)
>>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D
78DC 68DB 326D
>>>>> 8438
>>>>>>> --
>>>>>>> Olivier Sallou
>>>>>>> IRISA / University of Rennes 1
>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
>>>>>>> Tel: 02.99.84.71.95
>>>>>>>
>>>>>>> gpg key id: 4096R/326D8438  (keyring.debian.org)
>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D
>>> 8438

-- 
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95

gpg key id: 4096R/326D8438  (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438


Mime
View raw message