From yarn-issues-return-134348-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Tue Jan 9 01:02:05 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id E542F180607 for ; Tue, 9 Jan 2018 01:02:05 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D4B5D160C3E; Tue, 9 Jan 2018 00:02:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 21E4E160C2C for ; Tue, 9 Jan 2018 01:02:04 +0100 (CET) Received: (qmail 42316 invoked by uid 500); 9 Jan 2018 00:02:04 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 42305 invoked by uid 99); 9 Jan 2018 00:02:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jan 2018 00:02:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 8FB39C0230 for ; Tue, 9 Jan 2018 00:02:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.711 X-Spam-Level: X-Spam-Status: No, score=-100.711 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id LWYA4ePYqASj for ; Tue, 9 Jan 2018 00:02:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 1611A5FB87 for ; Tue, 9 Jan 2018 00:02:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 079F6E2582 for ; Tue, 9 Jan 2018 00:02:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 44BD12410E for ; Tue, 9 Jan 2018 00:02:00 +0000 (UTC) Date: Tue, 9 Jan 2018 00:02:00 +0000 (UTC) From: "Arun Suresh (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16317379#comment-16317379 ] Arun Suresh commented on YARN-6599: ----------------------------------- [~leftnoteasy], did a slightly deeper dive. This is my understanding of the general flow - I'd break this down the 2 phases. # When the app makes the allocate call: This is where you do the updatePendingAsk etc for each SchedulingRequest. This is also where you instantiate the AppPlacementAllocator and map it to the request. Looks like the really interesting method is {{validateAndSetSchedulingRequest}}, which is where you validate the placement constraints. This method sets the valid targetAllocationTags etc. # When the node heartbeats: At this point, in the leafQueue, we pick the SchedulingRequest with highest priority and finally, we call the {{canAllocate}} method which checks if the Node can be assigned to the scheduling request based on the placementConstratint. right ? Given the above, we should agree that: This approach - while it ensures that allocation of a SchedulingRequest to a node will guarantee that Constraints are NOT violated, It does nothing in the way of trying to FIND the nodes that will meet the constraints.. agreed ? My opinion - and something we should call out / document is that: * If an app is more concerned about priority ordering of its requests, then we can recommend using this approach. * If the app the more concerned about an optimal placement, then the processor route will give better results - since it activly tries to find nodes that satisfy constraints by considering multiple schedulingRequests and multiple nodes. Thuoghts ? Also some other comments: * In the Scheduler, you are adding a new List parameter to the allocate() method. Do you think, a better approach might be to create a common superclass / interface which both the SchedulingRequest and ResourceRequest extends and change the existing parameter to {{List}} ? * If we do the above, then you wont have to duplicate methods like : application.updateResourceRequests and normalizeSchedulingRequests > Support rich placement constraints in scheduler > ----------------------------------------------- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Wangda Tan > Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org