aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aurora ReviewBot <wfar...@apache.org>
Subject Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer
Date Fri, 06 May 2016 16:06:30 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review132056
-----------------------------------------------------------



Master (f25a4b1) is red with this patch.
  ./build-support/jenkins/build.sh

                           with temporary_dir() as checkpoint_root:
                             te = AuroraExecutor(
                     >           runner_provider=make_provider(checkpoint_root),
                                 sandbox_provider=DefaultTestSandboxProvider())
                     
                     src/test/python/apache/aurora/executor/test_thermos_executor.py:580:

                     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                     src/test/python/apache/aurora/executor/test_thermos_executor.py:193:
in make_provider
                         pex_location=thermos_runner_path(),
                     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                     
                     build = True
                     
                         def thermos_runner_path(build=True):
                           if not build:
                             return getattr(thermos_runner_path, 'value', None)
                         
                           if not hasattr(thermos_runner_path, 'value'):
                             pex_dir = safe_mkdtemp()
                     >       assert subprocess.call(["./pants", "--pants-distdir=%s"
% pex_dir, "binary",
                               "src/main/python/apache/thermos/runner:thermos_runner"])
== 0
                     E       assert 1 == 0
                     E        +  where 1 = <function call at 0x7f91bf83db18>(['./pants',
'--pants-distdir=/tmp/user/10021/tmpBf8mXY', 'binary', 'src/main/python/apache/thermos/runner:thermos_runner'])
                     E        +    where <function call at 0x7f91bf83db18>
= subprocess.call
                     
                     src/test/python/apache/aurora/executor/test_thermos_executor.py:185:
AssertionError
                     -------------- Captured stderr call --------------
                     Traceback (most recent call last):
                       File "/home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/0.0.80/bin/pants",
line 7, in <module>
                         from pants.bin.pants_exe import main
                     ImportError: No module named pants.bin.pants_exe
                      generated xml file: /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml

                      16 failed, 625 passed, 6 skipped, 1 warnings, 8 error in 140.62
seconds 
                     
FAILURE


               Waiting for background workers to finish.
16:06:25 03:14   [complete]
               FAILURE


I will refresh this build result if you post a review containing "@ReviewBot retry"

- Aurora ReviewBot


On May 6, 2016, 3:42 p.m., Joshua Cohen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> -----------------------------------------------------------
> 
> (Updated May 6, 2016, 3:42 p.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
>     https://issues.apache.org/jira/browse/AURORA-1636
>     https://issues.apache.org/jira/browse/AURORA-1637
>     https://issues.apache.org/jira/browse/AURORA-1638
>     https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to a bug in
parsing the docker_store_dir flag. Fixed here: https://reviews.apache.org/r/43451/ but has
not been backported to Mesos 0.27. This means we can only launch tasks that use AppC images
until we upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that launching
tasks with Docker images *does* work by using Aurora linked against 0.27.x but running Mesos
0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, but the
executor cannot setuid because the role-user does not exist), I've mounted /etc/passwd and
/etc/group into the container and added a new flag, `thermos_run_as_job_role`, to the scheduler.
This flag is only used when launching a task with a filesystem image, and causes us to add
`--execute-as-user <role from job key>` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points in the
filesystem from the image. It expects the full path to the mount to exist in the image. For
/etc/passwd and /etc/groups this is not a problem, but for the announcer acls file it was.
I ended up moving the announcer acl file into its own directory and mount that instead. In
conjunction with this I also had to modify our http_example Dockerfile to explicitly create
that mount point. A case could be made for sticking with the current path and just creating
an empty file in the image, I felt that creating an empty directory was slightly less gross.
This is tracked by https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running [docker2aci](https://github.com/appc/docker2aci)
on our http_example docker image. The base box needed to be upgraded to add this utility.
I haven't published the new base box yet even though I've updated the Vagrantfile to point
to version 6. Once this review has been approved and I'm sure there's no further changes that
need to be made I'll publish the base box before committing.
> 
> 
> Diffs
> -----
> 
>   3rdparty/python/requirements.txt 666c4ae487332f01380cfce76f0d97e2c6049c8e 
>   RELEASE-NOTES.md 8d5cbed2c627948c585241a8292a264e3d86120d 
>   Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 38470951e4482753fcada109ab12546a2fb146ce

>   build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
>   docs/features/containers.md 0d4791fb7b5ccae9c7b63341ab3dcff5bc62917c 
>   docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
>   docs/reference/scheduler-configuration.md 5f898a8d180cec6f3c02cb5b01673c56308ebd8a

>   examples/vagrant/announcer-auth.json  
>   examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_provisioner_backend PRE-CREATION

>   examples/vagrant/mesos_config/etc_mesos-slave/isolation PRE-CREATION 
>   examples/vagrant/upstart/aurora-scheduler.conf 084016abc169ed82b7ed00f5d14aea2e0ff38a49

>   src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java e1ce6380e08178e0cd6b1f1651e49c7a4337fb94

>   src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
32f2fa90b21189180e2bcd65a3cebf13f6551646 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java b325106c7f45b1ad1657221aaa39e3a428719ab0

>   src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java 9aadcebf547bd1eb4b4e238507e27ae2b699f473

>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 9eadf70240738030593a819a4a248588c9d1c35b

>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbContainer.java ae97638fa544dd8f8afbaa19b1dd31f5a1dc43d8

>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java a7523c4b258030bcfb2e457b083242ffa865a98a

>   src/main/python/apache/aurora/config/schema/base.py 00be8747d70dbf1cb370f09536588f8602d8fcce

>   src/main/python/apache/aurora/config/thrift.py 928ca9313b2c2062a322ba80b504a09c55e5377f

>   src/main/python/apache/aurora/executor/common/sandbox.py 36f1eabedc3ae47b23d9ab2ac0ab7a576ea36fd7

>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml cd181bb7ba0b3d725b51b1fe35303afa77294cb9

>   src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
98fe86000c2efa93a415824a680c1e144b0fac11 
>   src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ad397c6924b025f1eefb2bb02a6dc1e1f10ca078

>   src/test/java/org/apache/aurora/scheduler/storage/AbstractCronJobStoreTest.java 2343394a34541bc266f74b5da512cda6f8e6b56a

>   src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java af56115a4160e362d41bc9a571f5a1ca623ad199

>   src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java e43ec6cb35cbb454b967238dfb9ce006b21f4fb6

>   src/test/python/apache/aurora/config/test_thrift.py 7a076f0350ab2967abc6b8b7a2e5da0817926a56

>   src/test/python/apache/aurora/executor/common/test_sandbox.py bd402fc03c7790eab0198dd48414ad4de138e195

>   src/test/sh/org/apache/aurora/e2e/Dockerfile b2557b5a20cc369e31bd10ea92462bdb1879add7

>   src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 2813b6c79e4d44007dde79a10e2c7c9e9c1cecd9

>   src/test/sh/org/apache/aurora/e2e/http/http_example_bad_healthcheck.aurora 0534c9e589d10c53b834850477f95ad15b50010e

>   src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora b33e8f5cd95ce25ba0dc4c08da32783cecf1c44d

>   src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh eee6b4c62130567ecd5c32603feae88fce1c13a8

> 
> Diff: https://reviews.apache.org/r/46835/diff/
> 
> 
> Testing
> -------
> 
> ./gradlew build -Pq
> e2e tests with new base box.
> 
> 
> Thanks,
> 
> Joshua Cohen
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message