mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Peach (JIRA)" <>
Subject [jira] [Created] (MESOS-9300) XFS isolator can mislabel project IDs on persistence volumes.
Date Mon, 08 Oct 2018 18:39:00 GMT
James Peach created MESOS-9300:

             Summary: XFS isolator can mislabel project IDs on persistence volumes.
                 Key: MESOS-9300
             Project: Mesos
          Issue Type: Bug
          Components: agent
            Reporter: James Peach
            Assignee: James Peach

What happens here is that we are erroneously applying the sandbox's project ID to the persistent

First, the filesystem/linux isolator bind mounts the persistent volume into the sandbox:

I1003 06:49:21.907644 2812466 linux.cpp:593] Mounting '/srv/mesos/work/volumes/roles/pie.mobius/21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f'
to '/srv/mesos/work/slaves/909cff92-8e17-41bf-a251-9b5eb6186c35-S0/frameworks/363e6d80-8c38-46cf-815f-2fbf60a62628-0309/executors/mobius-mloop-1538549013_438156792-v2-shared-volume.pod1.writer-job.0.e93hs3uips2i9_1/runs/9e5770a7-9f78-46dc-9264-3e80be0e40cc/shared'
for persistent volume disk(allocated: pie.mobius)(reservations: [(DYNAMIC,pie.mobius,jarvis-principal,\{podInstance:
e93hs3uips2i9, pod: pod1, service: mobius-mloop-1538549013_438156792-v2-shared-volume})])[21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f:shared]<SHARED>:1
of container 9e5770a7-9f78-46dc-9264-3e80be0e40cc

Next, the `disk/xfs` isolator assigns a project ID to the sandbox:

I1003 06:49:21.920197 2812452 disk.cpp:402] Assigned project 6806 to '/srv/mesos/work/slaves/909cff92-8e17-41bf-a251-9b5eb6186c35-S0/frameworks/363e6d80-8c38-46cf-815f-2fbf60a62628-0309/executors/mobius-mloop-1538549013_438156792-v2-shared-volume.pod1.writer-job.0.e93hs3uips2i9_1/runs/9e5770a7-9f78-46dc-9264-3e80be0e40cc'

Note, that when this happens, the isolator recursively applies the project ID to the contents
of the sandbox. It doesn't follow symlinks or cross devices when it does this, but on Linux,
a bind mount would not trigger either of these conditions.

Finally, the `disk/xfs` isolator tries to assign a project ID to the persistent volume as
it is used by the task:

F1003 06:49:21.920577 2812452 disk.cpp:532] Check failed: scheduledProjects.contains(projectId.get())
untracked project ID 6806 for volume ID 21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f on /srv/mesos/work/volumes/roles/pie.mobius/21cb2eb6-b3e5-46f2-944e-8f6e5db9f07f

This check fails, because if the persistent volume has a project ID, we expect that is had
already be scheduled for reclaimation. However, it's project ID is the one we assigned to
the sandbox. We don't scheduled the ssandbox for reclaimation until cleanup, so (fortunately)
the invariant check triggers.

So, apart from triggering the CHECK, the root cause of this is that we are altering the project
ID of the persistent volume, which permanently misattributes the corresponding quote.

This message was sent by Atlassian JIRA

View raw message