Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 84AEA200C32 for ; Thu, 23 Feb 2017 00:31:49 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 8337A160B72; Wed, 22 Feb 2017 23:31:49 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 9F080160B62 for ; Thu, 23 Feb 2017 00:31:48 +0100 (CET) Received: (qmail 4832 invoked by uid 500); 22 Feb 2017 23:31:47 -0000 Mailing-List: contact reviews-help@aurora.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: reviews@aurora.apache.org Delivered-To: mailing list reviews@aurora.apache.org Received: (qmail 4810 invoked by uid 99); 22 Feb 2017 23:31:47 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Feb 2017 23:31:47 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 655B73131AB; Wed, 22 Feb 2017 23:31:47 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============8362764394800630803==" MIME-Version: 1.0 Subject: Re: Review Request 56690: [Patch 1/2] RFC for first patch implementing scheduling changes for Dynamic Reservations From: Stephan Erb To: Mehrdad Nurolahzade , Zameer Manji Cc: Dmitriy Shirchenko , Aurora Date: Wed, 22 Feb 2017 23:31:47 -0000 Message-ID: <20170222233147.1738.53592@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Stephan Erb X-ReviewGroup: Aurora X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/56690/ X-Sender: Stephan Erb References: <20170215014445.31018.87643@reviews.apache.org> In-Reply-To: <20170215014445.31018.87643@reviews.apache.org> Reply-To: Stephan Erb X-ReviewRequest-Repository: aurora archived-at: Wed, 22 Feb 2017 23:31:49 -0000 --===============8362764394800630803== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56690/#review166430 ----------------------------------------------------------- This is only a very partial review. My brain was not capable of more today :) src/main/java/org/apache/aurora/scheduler/TierInfo.java (lines 66 - 68) Copy paste error src/main/java/org/apache/aurora/scheduler/TierInfo.java (lines 74 - 81) A mutable `TierConfig` makes the code very hard to reason about, especially in the context of multi-threading. Do you see a way to keep the `TierConfig` immutable at runtime? src/main/java/org/apache/aurora/scheduler/TierManager.java (lines 100 - 105) In addition to my concerns about the mutable tier config, the `copyOf` might come with a performance tax. `getTier` is on a hot code path and I therefore recently rewrote the `checkArgument` here to only construct the String if the argument is invalid. This yielded a significant perf improvement in our scheduling benchmarks. src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java (lines 51 - 63) `IAssignedTask` has a `task` which in turn has a jobkey. There should therfore be no need to parse this from the Mesos `TaskInfo`. src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java (lines 405 - 406) Please add a doc string. I may not have completely understood why this is absolutely needed. We should have a very good reason to break the existing interface. src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java (lines 99 - 100) To short-circuit both loops this should probably be a return. src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java (line 166) Please make sure you always use the built-in formatting of the logger methods. Otherwise we will assemble strings in memory, only to be discarded by the logger afterwards :-) src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java (line 208) Same here (and in some other places) src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java (lines 210 - 228) This reads like as if there is a more general concept of "soft-constraints" waiting to be discovered. In other words: We want constraints that are lifted once we cannot satisfy them for a certain amount of time. See https://issues.apache.org/jira/browse/AURORA-173 for details. I imagine the TaskAssigner could keep its current behavior and only assign a task if there is no veto for it. It is the responsibility of an out-of-band mechanism to ensure we either don't generate vetos for expired soft-constraints, or we have a way to check if a generated veto is already expired or not. (One very rudimentary implementation could probably look similar to `TaskTimeout.java` but for the `PENDING` state. However there are most likely also other/better ways to do this) src/main/resources/org/apache/aurora/scheduler/tiers.json (lines 29 - 33) Shoudn't reservations be always backed by quota to prevent abuse? - Stephan Erb On Feb. 15, 2017, 2:44 a.m., Dmitriy Shirchenko wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56690/ > ----------------------------------------------------------- > > (Updated Feb. 15, 2017, 2:44 a.m.) > > > Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji. > > > Bugs: AURORA-1819 > https://issues.apache.org/jira/browse/AURORA-1819 > > > Repository: aurora > > > Description > ------- > > This is an RFC (without tests) for dynamic reservations proposal. If there is consensus on the approach, I will add tests. This patch was also tested locally and works as expected. > > > Diffs > ----- > > src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java f2296a9d7a88be7e43124370edecfe64415df00f > src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 6f2ca35c5d83dde29c24865b4826d4932e96da80 > src/main/java/org/apache/aurora/scheduler/TaskVars.java 676dfd9f9d7ee0633c05424f788fd0ab116976bb > src/main/java/org/apache/aurora/scheduler/TierInfo.java c45b949ae7946fc92d7e62f94696ddc4f0790cfa > src/main/java/org/apache/aurora/scheduler/TierManager.java c6ad2b1c48673ca2c14ddd308684d81ce536beca > src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java b12ac83168401c15fb1d30179ea8e4816f09cd3d > src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 > src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java f0b148cd158d61cd89cc51dca9f3fa4c6feb1b49 > src/main/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilter.java f6c759f03c4152ae93317692fc9db202fe251122 > src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java bb1a960a4c77f48b0ceaa213bd27546551f384f9 > src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 60097d91d836e2686d6e90571f13a2fbfd88ae14 > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 0d639f66db456858278b0485c91c40975c3b45ac > src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 8c000cb0626bd34f6f30e23fe2b3a045f2b44e35 > src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java e16e36ed360ef9ca371df9084365ea88cfb6e7ce > src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 202cae96ffc5b49e638b973a273f7983137b5baf > src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 9aa263a9cfae03a9a0c5bc7fe3a1405397d3009c > src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java 203f62bacc47470545d095e4d25f7e0f25990ed9 > src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java da378e84ee65a658ff2382489d3ab6d5f6451b5f > src/main/resources/org/apache/aurora/scheduler/tiers.json 34ddb1dc769a73115c209c9b2ee158cd364392d8 > src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 82e40d509d84c37a19b6a9ef942283d908833840 > src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 49d4e82cc03144b80292fe43066a6cc4d7aed88f > src/test/java/org/apache/aurora/scheduler/resources/AcceptedOfferTest.java dded9c34749cf599d197ed312ffb6bf63b6033f1 > src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java b8b8edb1a21ba89b8b60f8f8451c8c776fc23ae8 > src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java e04f6113c43eca4555ee0719f8208d7c4ebb8d61 > src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java cf2d25ec2e407df7159e0021ddb44adf937e1777 > > Diff: https://reviews.apache.org/r/56690/diff/ > > > Testing > ------- > > > Thanks, > > Dmitriy Shirchenko > > --===============8362764394800630803==--