mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Rukletsov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-6653) Overlayfs backend may fail to mount the rootfs if both container image and image volume are specified.
Date Tue, 07 Feb 2017 18:17:41 GMT

    [ https://issues.apache.org/jira/browse/MESOS-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856464#comment-15856464
] 

Alexander Rukletsov commented on MESOS-6653:
--------------------------------------------

[~gilbert], [~jieyu], [~tillt] I'm retargeting this for 1.1.2. Please either backport it to
1.1.x or remove the target version.

> Overlayfs backend may fail to mount the rootfs if both container image and image volume
are specified.
> ------------------------------------------------------------------------------------------------------
>
>                 Key: MESOS-6653
>                 URL: https://issues.apache.org/jira/browse/MESOS-6653
>             Project: Mesos
>          Issue Type: Bug
>          Components: containerization
>            Reporter: Gilbert Song
>            Assignee: Gilbert Song
>            Priority: Critical
>              Labels: backend, containerizer, overlayfs
>             Fix For: 1.2.0
>
>
> Depending on MESOS-6000, we use symlink to shorten the overlayfs mounting arguments.
However, if more than one image need to be provisioned (e.g., a container image is specified
while image volumes are specified for the same container), the symlink .../backends/overlay/links
would fail to be created since it exists already.
> Here is a simple log when we hard code overlayfs as our default backend:
> {noformat}
> [07:02:45] :	 [Step 10/10] [ RUN      ] Nesting/VolumeImageIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem/0
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.416021  2919 containerizer.cpp:207] Using isolation:
filesystem/linux,volume/image,docker/runtime,network/cni
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.419312  2919 linux_launcher.cpp:150] Using
/sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
> [07:02:46] :	 [Step 10/10] E1127 07:02:46.425336  2919 shell.hpp:107] Command 'hadoop
version 2>&1' failed; this is the output:
> [07:02:46] :	 [Step 10/10] sh: 1: hadoop: not found
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.425379  2919 fetcher.cpp:69] Skipping URI fetcher
plugin 'hadoop' as it could not be created: Failed to create HDFS client: Failed to execute
'hadoop version 2>&1'; the command was either not found or exited with a non-zero exit
status: 127
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.425452  2919 local_puller.cpp:94] Creating
local puller with docker registry '/tmp/R6OUei/registry'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.427258  2934 containerizer.cpp:956] Starting
container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 for executor 'test_executor' of framework 
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.427592  2938 metadata_manager.cpp:167] Looking
for image 'test_image_rootfs'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.427774  2936 local_puller.cpp:147] Untarring
image 'test_image_rootfs' from '/tmp/R6OUei/registry/test_image_rootfs.tar' to '/tmp/R6OUei/store/staging/9krDz2'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.512070  2933 local_puller.cpp:167] The repositories
JSON file for image 'test_image_rootfs' is '{"test_image_rootfs":{"latest":"815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346"}}'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.512279  2933 local_puller.cpp:295] Extracting
layer tar ball '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/layer.tar
to rootfs '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/rootfs'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.617442  2937 metadata_manager.cpp:155] Successfully
cached image 'test_image_rootfs'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.617908  2938 provisioner.cpp:286] Image layers:
1
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.617925  2938 provisioner.cpp:296] Should hit
here
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.617949  2938 provisioner.cpp:315] !!!!: bind
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.617959  2938 provisioner.cpp:315] !!!!: overlay
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.617967  2938 provisioner.cpp:315] !!!!: copy
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.617974  2938 provisioner.cpp:318] Provisioning
image rootfs '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/rootfses/c71e83d2-5dbe-4eb7-a2fc-b8cc826771f7'
for container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 using overlay backend
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.618408  2936 overlay.cpp:175] Created symlink
'/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/links'
-> '/tmp/DQ3blT'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.618472  2936 overlay.cpp:203] Provisioning
image rootfs with overlayfs: 'lowerdir=/tmp/DQ3blT/0,upperdir=/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/scratch/c71e83d2-5dbe-4eb7-a2fc-b8cc826771f7/upperdir,workdir=/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/scratch/c71e83d2-5dbe-4eb7-a2fc-b8cc826771f7/workdir'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.619098  2933 linux.cpp:451] Ignored an image
volume for container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.619745  2938 metadata_manager.cpp:167] Looking
for image 'test_image_volume'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.619925  2937 local_puller.cpp:147] Untarring
image 'test_image_volume' from '/tmp/R6OUei/registry/test_image_volume.tar' to '/tmp/R6OUei/store/staging/2GNlJO'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.713526  2935 local_puller.cpp:167] The repositories
JSON file for image 'test_image_volume' is '{"test_image_volume":{"latest":"815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346"}}'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.713726  2935 local_puller.cpp:295] Extracting
layer tar ball '/tmp/R6OUei/store/staging/2GNlJO/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/layer.tar
to rootfs '/tmp/R6OUei/store/staging/2GNlJO/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/rootfs'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.818696  2937 metadata_manager.cpp:155] Successfully
cached image 'test_image_volume'
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.819169  2934 provisioner.cpp:286] Image layers:
1
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.819188  2934 provisioner.cpp:296] Should hit
here
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.819221  2934 provisioner.cpp:315] !!!!: bind
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.819232  2934 provisioner.cpp:315] !!!!: overlay
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.819236  2934 provisioner.cpp:315] !!!!: copy
> [07:02:46] :	 [Step 10/10] I1127 07:02:46.819241  2934 provisioner.cpp:318] Provisioning
image rootfs '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/rootfses/baf632b3-29c5-45e4-9d2e-6f3a2bdd9759'
for container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 using overlay backend
> [07:02:46] :	 [Step 10/10] ../../src/tests/containerizer/volume_image_isolator_tests.cpp:214:
Failure
> [07:02:46] :	 [Step 10/10] (launch).failure(): Failed to create symlink '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/links'
-> '/tmp/6dj9IG'
> [07:02:46] :	 [Step 10/10] [  FAILED  ] Nesting/VolumeImageIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem/0,
where GetParam() = false (919 ms)
> {noformat}
> We should differenciate the links for different provisioned images.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message