Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 11132200D3C for ; Tue, 14 Nov 2017 23:41:57 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 0F7D1160C09; Tue, 14 Nov 2017 22:41:57 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 37566160BF4 for ; Tue, 14 Nov 2017 23:41:56 +0100 (CET) Received: (qmail 53754 invoked by uid 500); 14 Nov 2017 22:41:51 -0000 Mailing-List: contact reviews-help@aurora.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: reviews@aurora.apache.org Delivered-To: mailing list reviews@aurora.apache.org Received: (qmail 53191 invoked by uid 99); 14 Nov 2017 22:41:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Nov 2017 22:41:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 47FA51807C5; Tue, 14 Nov 2017 22:41:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.251 X-Spam-Level: *** X-Spam-Status: No, score=3.251 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, KAM_LOTSOFHASH=0.25, RP_MATCHES_RCVD=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id GuUC-57ck-cx; Tue, 14 Nov 2017 22:41:48 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id A188F5FD47; Tue, 14 Nov 2017 22:41:47 +0000 (UTC) Received: from reviews.apache.org (unknown [10.41.0.12]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 810E3E00C9; Tue, 14 Nov 2017 22:41:46 +0000 (UTC) Received: from reviews-vm2.apache.org (localhost [IPv6:::1]) by reviews.apache.org (ASF Mail Server at reviews-vm2.apache.org) with ESMTP id 2517AC401B6; Tue, 14 Nov 2017 22:41:46 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============5351160647694637663==" MIME-Version: 1.0 Subject: Re: Review Request 63536: Give jobs the ability to determine how to handle partitions by integrating with new Mesos Partition-Aware APIs From: David McLaughlin To: Bill Farner , Santhosh Kumar Shanmugham , Jordan Ly Cc: Aurora , Stephan Erb , Aurora ReviewBot , David McLaughlin Date: Tue, 14 Nov 2017 22:41:46 -0000 Message-ID: <20171114224146.16898.17717@reviews-vm2.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: David McLaughlin X-ReviewGroup: Aurora X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/63536/ X-Sender: David McLaughlin References: <20171110234530.31650.84952@reviews-vm2.apache.org> In-Reply-To: <20171110234530.31650.84952@reviews-vm2.apache.org> X-ReviewBoard-Diff-For: src/test/java/org/apache/aurora/scheduler/state/PartitionManagerTest.java X-ReviewBoard-Diff-For: src/main/java/org/apache/aurora/scheduler/state/PartitionManager.java Reply-To: David McLaughlin X-ReviewRequest-Repository: aurora archived-at: Tue, 14 Nov 2017 22:41:57 -0000 --===============5351160647694637663== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63536/ ----------------------------------------------------------- (Updated Nov. 14, 2017, 10:41 p.m.) Review request for Aurora, Jordan Ly, Santhosh Kumar Shanmugham, and Bill Farner. Changes ------- Rebase + green build. Server side is code-complete, but I'll follow up with a change to the Python DSL to match the recommendation in the proposal document. Repository: aurora Description ------- This is my prototype code for adding partition-awareness to Aurora. There is a proposal document to accompany this here: https://docs.google.com/document/d/1E3GlsVTJLEMAkDWk2_PTxzkRZcapb8nF_5q5AADQI7g/edit# I'd like feedback on the high-level approach before adding unit tests, metrics, logging, etc. Diffs (updated) ----- api/src/main/thrift/org/apache/aurora/gen/api.thrift 1d369263d779b549b9304018437f535ddc855966 examples/vagrant/upstart/aurora-scheduler.conf 5ca3caef03b6632cd4dbf47711b1ef183f6a6449 src/main/java/org/apache/aurora/scheduler/base/Conversions.java 33cc012a2cad929b0dd1ce236597b870cfc5aba0 src/main/java/org/apache/aurora/scheduler/base/Jobs.java 8d5f4e57c6b4f847cb74471f246fd0b7dd0cbc36 src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 7c223eae69503fe1d5bf34c430438637abcbcb9b src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java 5e83b2acdde7198d16427d4031e9772f78612554 src/main/java/org/apache/aurora/scheduler/state/PartitionManager.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/state/SideEffect.java b91a0852e968b4aa9d74801601671cb61af3648a src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 9989ed441cd9bc442e6472768880ce7924c3bdd9 src/main/java/org/apache/aurora/scheduler/state/StateModule.java c03fff11ea3a4086f9daaa8b07315006c1b481e4 src/main/java/org/apache/aurora/scheduler/state/TaskStateMachine.java eb4fe7d78ad1e6ec430c428df527bd0cf3a053c1 src/main/python/apache/aurora/client/cli/jobs.py b79ae56bee0e5692cacf1e66f4a4126b06aaffdc src/main/python/apache/aurora/config/schema/base.py 18ce826363009e1cc0beac5cce4edf42610d487e src/main/python/apache/aurora/config/thrift.py bedf8cd6641e1b1a930602791b758d584af4891c src/test/java/org/apache/aurora/scheduler/base/JobsTest.java 13f656f241a8a9a3d339f4053f165070c2669ef3 src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java c2d875bb5c393dd95d75251fe86dc649ceba7bd9 src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java 7b0429109e9a7795e559db264e7737fc55ff0169 src/test/java/org/apache/aurora/scheduler/state/PartitionManagerTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java 0366cd6e9ddba0c3b9c88ffb50738767a352a17a src/test/java/org/apache/aurora/scheduler/state/TaskStateMachineTest.java 8d6c3fff0af2df39bb929f760b862a2edf5d6fca src/test/python/apache/aurora/client/cli/test_task.py 186cb2737ba8e169819b7d54f86a7344a669b6cb Diff: https://reviews.apache.org/r/63536/diff/4/ Changes: https://reviews.apache.org/r/63536/diff/3-4/ Testing ------- Manual testing in Vagrant by stopping and starting the Mesos agent. With three jobs: 1) No PartitionPolicy (verified existing behavior of moving from PARTITIONED directly to LOST) 2) PartitionPolicy with custom delay_secs (verified sat in PARTITIONED for a while before moving to LOST) 3) PartitionPolicy with reschedule=False (verified sat in PARTITIONED indefinitely) I also verified tasks are able to transition to various states (back to RUNNING or moving to FAILED, etc.) when you turn the agent back on. File Attachments ---------------- Task in PARTITIONED state https://reviews.apache.org/media/uploaded/files/2017/11/07/02c7fc72-b11d-4ef9-a86b-914e748cad99__Screen_Shot_2017-11-07_at_11.23.41_AM.png Task back into running when partition resolved https://reviews.apache.org/media/uploaded/files/2017/11/07/a0413f54-1572-4410-a386-0a22e78fab13__Screen_Shot_2017-11-07_at_11.26.02_AM.png Compaction of PARTITIONED cycles (note timestamps) https://reviews.apache.org/media/uploaded/files/2017/11/07/edec32e5-b3ec-4fdc-b93f-5449519805ae__Screen_Shot_2017-11-07_at_11.27.47_AM.png Thanks, David McLaughlin --===============5351160647694637663==--