Return-Path: X-Original-To: apmail-aurora-reviews-archive@minotaur.apache.org Delivered-To: apmail-aurora-reviews-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E1CE3179A0 for ; Sun, 22 Mar 2015 03:52:03 +0000 (UTC) Received: (qmail 95028 invoked by uid 500); 22 Mar 2015 03:52:03 -0000 Delivered-To: apmail-aurora-reviews-archive@aurora.apache.org Received: (qmail 94983 invoked by uid 500); 22 Mar 2015 03:52:03 -0000 Mailing-List: contact reviews-help@aurora.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: reviews@aurora.incubator.apache.org Delivered-To: mailing list reviews@aurora.incubator.apache.org Received: (qmail 94971 invoked by uid 99); 22 Mar 2015 03:52:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Mar 2015 03:52:03 +0000 X-ASF-Spam-Status: No, hits=-1997.8 required=5.0 tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Sun, 22 Mar 2015 03:52:01 +0000 Received: (qmail 94939 invoked by uid 99); 22 Mar 2015 03:51:41 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Mar 2015 03:51:41 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id C32571D466E; Sun, 22 Mar 2015 03:51:40 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============4615770620330448736==" MIME-Version: 1.0 Subject: Review Request 32373: Add pesos executor build target From: "Brian Wickman" To: "Bill Farner" , "Kevin Sweeney" Cc: "Aurora" , "Brian Wickman" Date: Sun, 22 Mar 2015 03:51:40 -0000 Message-ID: <20150322035140.1489.6458@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Brian Wickman" X-ReviewGroup: Aurora X-ReviewRequest-URL: https://reviews.apache.org/r/32373/ X-Sender: "Brian Wickman" Reply-To: "Brian Wickman" X-ReviewRequest-Repository: aurora X-Virus-Checked: Checked by ClamAV on apache.org --===============4615770620330448736== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32373/ ----------------------------------------------------------- Review request for Aurora, Kevin Sweeney and Bill Farner. Bugs: AURORA-1031 https://issues.apache.org/jira/browse/AURORA-1031 Repository: aurora Description ------- 1. Add a level of indirection for importing mesos_pb2.py. The mesos_pb2 stub that comes with mesos.interface is compiled with protobuf 2.5, but in order for pesos to work with python 3.x, we must use protobuf 2.6, and these two are not code-compatible with each other (they are wire-compatible however.) 2. Add a build target for a pesos-enabled executor. pesos is definitely alpha-level code but the end-to-end tests work. This gives us a harness to do better integration testing. The only thing I haven't tested is slave disconnection / reconnection / framework reregistration. Thankfully the executor side of the protocol is much simpler than the scheduler side, so this should be easy to test in pesos w/o an integration test. Diffs ----- 3rdparty/python/BUILD 7ef81d13afa0089d7e8a779e71b53e0fc1848466 examples/vagrant/aurorabuild.sh 7dc9777c054a8f8a98059f263c49ba2e3e6d7207 src/main/python/apache/aurora/executor/BUILD cbb2f5f7b5daa936db71cf8c0aac8ddb2002060b src/main/python/apache/aurora/executor/aurora_executor.py df0df0cba3269a137a370e2b4096fd61fd3af258 src/main/python/apache/aurora/executor/bin/BUILD 25cc3d00ce7548ab54f1a2b857a3e6dbb1a85201 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 3174e1f9635c47b2f3d789c39b1bc177db25a7e5 src/main/python/apache/aurora/executor/common/BUILD 3d94792b9e3c2d0f7bb7ce42616f08947f5119ee src/main/python/apache/aurora/executor/common/announcer.py dda76f018f472d7d8228459eb89f4c5daf9df26d src/main/python/apache/aurora/executor/common/health_checker.py cfc29c3164837493e4ddfb8314bb6999ddaa0c91 src/main/python/apache/aurora/executor/common/kill_manager.py 23172676fc6222f2a8cab1398118e1b8b550c3ff src/main/python/apache/aurora/executor/common/resource_manager.py b7dc40d8973ec2e5998ab4f6ff988051a70bb1ab src/main/python/apache/aurora/executor/common/status_checker.py 795dae2d6b661fc528d952c2315196d94127961f src/main/python/apache/aurora/executor/executor_base.py 1db97cc8c12752d4eca339a7680ba963a66ffbce src/main/python/apache/aurora/executor/gc_executor.py a7776b599f5fc028ec2ce7712856e080381e84a6 src/main/python/apache/aurora/executor/thermos_task_runner.py 505a1e69ebfe1bd2fa74623063cb923ecff8a021 src/test/python/apache/aurora/executor/BUILD 013f056d2e131ae8077f0fcc542a340195a90e50 src/test/python/apache/aurora/executor/common/BUILD b3da27bb83c44a228851b256cc09f093d1e2eb87 src/test/python/apache/aurora/executor/common/test_health_checker.py 1b4423a1eb95cc950206a355fa657c8082c8d93f src/test/python/apache/aurora/executor/common/test_kill_manager.py f3d43af4e707cf22bfbc5499c35b7deeb0ea7f43 src/test/python/apache/aurora/executor/common/test_resource_manager.py a898e4d81d34d1e30e39db1be1a66bc9e0ab1a35 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py c473808d3df6cfd44c3515f2b6dcf20b0efa1075 src/test/python/apache/aurora/executor/common/test_status_checker.py 5be1981c8c8e88258456adb21aa3ca7c0aa472a7 src/test/python/apache/aurora/executor/test_executor_base.py e4135cecb8af0dd4749146d3f3671b3d576f8265 src/test/python/apache/aurora/executor/test_gc_executor.py 0f0b4a3781295c2a6640d272f9c8b0e8f78cc8a8 src/test/python/apache/aurora/executor/test_status_manager.py ce4679ba1aa7b42cf0115c943d84663030182d23 src/test/python/apache/aurora/executor/test_thermos_executor.py 26b26f452fb7c718e4f99f470211dfe926ebe354 src/test/python/apache/aurora/executor/test_thermos_task_runner.py 6b24bbb2ab7ca16f97961aabeed945b61e5b5908 Diff: https://reviews.apache.org/r/32373/diff/ Testing ------- ./pants test src/test/python/apache/aurora/executor:: Ran e2e tests with the pesos executor using pesos==0.2.0-dev0 published into third_party/ from the wickman/helpers branch here: https://github.com/wickman/pesos/tree/wickman/helpers pesos executor log: https://gist.github.com/wickman/8640478fcaa660f23740 pesos executor dependencies (no mesos.native, yay): mba=aurora=; zipinfo -1 dist/thermos_executor_pesos.pex | grep .deps | cut -d/ -f2 | sort -u PyYAML-3.11-cp27-none-macosx_10_9_intel.whl api.src.main.thrift.org.apache.aurora.gen.py_thrift_342086a0-0.0.0-py2-none-any.whl api.src.main.thrift.org.apache.aurora.gen.py_thrift_storage_a5536062-0.0.0-py2-none-any.whl api.src.main.thrift.org.apache.aurora.gen.py_thrift_test_d991619f-0.0.0-py2-none-any.whl api.src.main.thrift.org.apache.thermos.py_thrift_b45199dc-0.0.0-py2-none-any.whl argh-0.26.1-py2-none-any.whl backports.ssl_match_hostname-3.4.0.2-py2-none-any.whl certifi-14.05.14-py2.py3-none-any.whl compactor-0.2.0-py2.py3-none-any.whl futures-2.1.6-py2.py3-none-any.whl kazoo-1.3.1-py2-none-any.whl mesos.interface-0.21.1-py2-none-any.whl pathtools-0.1.2-py2-none-any.whl pesos-0.2.0_dev0-py2.py3-none-any.whl pex-0.8.2-py2.py3-none-any.whl protobuf-2.6.1-py2.7.egg psutil-2.1.3-cp27-none-macosx_10_9_intel.whl pystachio-0.7.4-py2.py3-none-any.whl requests-2.3.0-py2.py3-none-any.whl setuptools-7.0-py2.py3-none-any.whl thrift-0.9.1-cp27-none-macosx_10_9_intel.whl tornado-4.1-cp27-none-macosx_10_9_intel.whl trollius-1.0.4-py27-none-any.whl twitter.common.app-0.3.3-py2-none-any.whl twitter.common.collections-0.3.3-py2-none-any.whl twitter.common.concurrent-0.3.3-py2-none-any.whl twitter.common.contextutil-0.3.3-py2-none-any.whl twitter.common.decorators-0.3.3-py2-none-any.whl twitter.common.dirutil-0.3.3-py2-none-any.whl twitter.common.exceptions-0.3.3-py2-none-any.whl twitter.common.lang-0.3.3-py2-none-any.whl twitter.common.log-0.3.3-py2-none-any.whl twitter.common.metrics-0.3.3-py2-none-any.whl twitter.common.options-0.3.3-py2-none-any.whl twitter.common.process-0.3.3-py2-none-any.whl twitter.common.quantity-0.3.3-py2-none-any.whl twitter.common.recordio-0.3.3-py2-none-any.whl twitter.common.string-0.3.3-py2-none-any.whl twitter.common.util-0.3.3-py2-none-any.whl twitter.common.zookeeper-0.3.3-py2-none-any.whl watchdog-0.7.0-cp27-none-macosx_10_9_intel.whl zope.interface-4.1.2-cp27-none-macosx_10_9_intel.whl Thanks, Brian Wickman --===============4615770620330448736==--