aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Wickman" <wick...@apache.org>
Subject Review Request 32373: Add pesos executor build target
Date Sun, 22 Mar 2015 03:51:40 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32373/
-----------------------------------------------------------

Review request for Aurora, Kevin Sweeney and Bill Farner.


Bugs: AURORA-1031
    https://issues.apache.org/jira/browse/AURORA-1031


Repository: aurora


Description
-------

1.  Add a level of indirection for importing mesos_pb2.py.  The mesos_pb2 stub that comes
with mesos.interface is compiled with protobuf 2.5, but in order for pesos to work with python
3.x, we must use protobuf 2.6, and these two are not code-compatible with each other (they
are wire-compatible however.)
2.  Add a build target for a pesos-enabled executor.

pesos is definitely alpha-level code but the end-to-end tests work.  This gives us a harness
to do better integration testing.  The only thing I haven't tested is slave disconnection
/ reconnection / framework reregistration.  Thankfully the executor side of the protocol is
much simpler than the scheduler side, so this should be easy to test in pesos w/o an integration
test.


Diffs
-----

  3rdparty/python/BUILD 7ef81d13afa0089d7e8a779e71b53e0fc1848466 
  examples/vagrant/aurorabuild.sh 7dc9777c054a8f8a98059f263c49ba2e3e6d7207 
  src/main/python/apache/aurora/executor/BUILD cbb2f5f7b5daa936db71cf8c0aac8ddb2002060b 
  src/main/python/apache/aurora/executor/aurora_executor.py df0df0cba3269a137a370e2b4096fd61fd3af258

  src/main/python/apache/aurora/executor/bin/BUILD 25cc3d00ce7548ab54f1a2b857a3e6dbb1a85201

  src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 3174e1f9635c47b2f3d789c39b1bc177db25a7e5

  src/main/python/apache/aurora/executor/common/BUILD 3d94792b9e3c2d0f7bb7ce42616f08947f5119ee

  src/main/python/apache/aurora/executor/common/announcer.py dda76f018f472d7d8228459eb89f4c5daf9df26d

  src/main/python/apache/aurora/executor/common/health_checker.py cfc29c3164837493e4ddfb8314bb6999ddaa0c91

  src/main/python/apache/aurora/executor/common/kill_manager.py 23172676fc6222f2a8cab1398118e1b8b550c3ff

  src/main/python/apache/aurora/executor/common/resource_manager.py b7dc40d8973ec2e5998ab4f6ff988051a70bb1ab

  src/main/python/apache/aurora/executor/common/status_checker.py 795dae2d6b661fc528d952c2315196d94127961f

  src/main/python/apache/aurora/executor/executor_base.py 1db97cc8c12752d4eca339a7680ba963a66ffbce

  src/main/python/apache/aurora/executor/gc_executor.py a7776b599f5fc028ec2ce7712856e080381e84a6

  src/main/python/apache/aurora/executor/thermos_task_runner.py 505a1e69ebfe1bd2fa74623063cb923ecff8a021

  src/test/python/apache/aurora/executor/BUILD 013f056d2e131ae8077f0fcc542a340195a90e50 
  src/test/python/apache/aurora/executor/common/BUILD b3da27bb83c44a228851b256cc09f093d1e2eb87

  src/test/python/apache/aurora/executor/common/test_health_checker.py 1b4423a1eb95cc950206a355fa657c8082c8d93f

  src/test/python/apache/aurora/executor/common/test_kill_manager.py f3d43af4e707cf22bfbc5499c35b7deeb0ea7f43

  src/test/python/apache/aurora/executor/common/test_resource_manager.py a898e4d81d34d1e30e39db1be1a66bc9e0ab1a35

  src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py c473808d3df6cfd44c3515f2b6dcf20b0efa1075

  src/test/python/apache/aurora/executor/common/test_status_checker.py 5be1981c8c8e88258456adb21aa3ca7c0aa472a7

  src/test/python/apache/aurora/executor/test_executor_base.py e4135cecb8af0dd4749146d3f3671b3d576f8265

  src/test/python/apache/aurora/executor/test_gc_executor.py 0f0b4a3781295c2a6640d272f9c8b0e8f78cc8a8

  src/test/python/apache/aurora/executor/test_status_manager.py ce4679ba1aa7b42cf0115c943d84663030182d23

  src/test/python/apache/aurora/executor/test_thermos_executor.py 26b26f452fb7c718e4f99f470211dfe926ebe354

  src/test/python/apache/aurora/executor/test_thermos_task_runner.py 6b24bbb2ab7ca16f97961aabeed945b61e5b5908


Diff: https://reviews.apache.org/r/32373/diff/


Testing
-------

./pants test src/test/python/apache/aurora/executor::

Ran e2e tests with the pesos executor using pesos==0.2.0-dev0 published into third_party/
from the wickman/helpers branch here: https://github.com/wickman/pesos/tree/wickman/helpers

pesos executor log: https://gist.github.com/wickman/8640478fcaa660f23740

pesos executor dependencies (no mesos.native, yay):

mba=aurora=; zipinfo -1 dist/thermos_executor_pesos.pex  | grep .deps | cut -d/ -f2 | sort
-u
PyYAML-3.11-cp27-none-macosx_10_9_intel.whl
api.src.main.thrift.org.apache.aurora.gen.py_thrift_342086a0-0.0.0-py2-none-any.whl
api.src.main.thrift.org.apache.aurora.gen.py_thrift_storage_a5536062-0.0.0-py2-none-any.whl
api.src.main.thrift.org.apache.aurora.gen.py_thrift_test_d991619f-0.0.0-py2-none-any.whl
api.src.main.thrift.org.apache.thermos.py_thrift_b45199dc-0.0.0-py2-none-any.whl
argh-0.26.1-py2-none-any.whl
backports.ssl_match_hostname-3.4.0.2-py2-none-any.whl
certifi-14.05.14-py2.py3-none-any.whl
compactor-0.2.0-py2.py3-none-any.whl
futures-2.1.6-py2.py3-none-any.whl
kazoo-1.3.1-py2-none-any.whl
mesos.interface-0.21.1-py2-none-any.whl
pathtools-0.1.2-py2-none-any.whl
pesos-0.2.0_dev0-py2.py3-none-any.whl
pex-0.8.2-py2.py3-none-any.whl
protobuf-2.6.1-py2.7.egg
psutil-2.1.3-cp27-none-macosx_10_9_intel.whl
pystachio-0.7.4-py2.py3-none-any.whl
requests-2.3.0-py2.py3-none-any.whl
setuptools-7.0-py2.py3-none-any.whl
thrift-0.9.1-cp27-none-macosx_10_9_intel.whl
tornado-4.1-cp27-none-macosx_10_9_intel.whl
trollius-1.0.4-py27-none-any.whl
twitter.common.app-0.3.3-py2-none-any.whl
twitter.common.collections-0.3.3-py2-none-any.whl
twitter.common.concurrent-0.3.3-py2-none-any.whl
twitter.common.contextutil-0.3.3-py2-none-any.whl
twitter.common.decorators-0.3.3-py2-none-any.whl
twitter.common.dirutil-0.3.3-py2-none-any.whl
twitter.common.exceptions-0.3.3-py2-none-any.whl
twitter.common.lang-0.3.3-py2-none-any.whl
twitter.common.log-0.3.3-py2-none-any.whl
twitter.common.metrics-0.3.3-py2-none-any.whl
twitter.common.options-0.3.3-py2-none-any.whl
twitter.common.process-0.3.3-py2-none-any.whl
twitter.common.quantity-0.3.3-py2-none-any.whl
twitter.common.recordio-0.3.3-py2-none-any.whl
twitter.common.string-0.3.3-py2-none-any.whl
twitter.common.util-0.3.3-py2-none-any.whl
twitter.common.zookeeper-0.3.3-py2-none-any.whl
watchdog-0.7.0-cp27-none-macosx_10_9_intel.whl
zope.interface-4.1.2-cp27-none-macosx_10_9_intel.whl


Thanks,

Brian Wickman


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message