From yarn-issues-return-135521-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Mon Jan 22 21:05:04 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id D1FA6180609 for ; Mon, 22 Jan 2018 21:05:04 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C2792160C4B; Mon, 22 Jan 2018 20:05:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 14EDE160C32 for ; Mon, 22 Jan 2018 21:05:03 +0100 (CET) Received: (qmail 39571 invoked by uid 500); 22 Jan 2018 20:05:03 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 39560 invoked by uid 99); 22 Jan 2018 20:05:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jan 2018 20:05:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 904F9C3BE9 for ; Mon, 22 Jan 2018 20:05:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -107.911 X-Spam-Level: X-Spam-Status: No, score=-107.911 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 0mGuRReVAicI for ; Mon, 22 Jan 2018 20:05:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id A50F85FAC8 for ; Mon, 22 Jan 2018 20:05:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 35724E0F78 for ; Mon, 22 Jan 2018 20:05:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 95C30241C4 for ; Mon, 22 Jan 2018 20:05:00 +0000 (UTC) Date: Mon, 22 Jan 2018 20:05:00 +0000 (UTC) From: "Arun Suresh (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (YARN-7783) Add validation step to ensure constraints are not violated due to order in which a request is processed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-7783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-7783: ------------------------------ Attachment: YARN-7783-YARN-6592.003.patch > Add validation step to ensure constraints are not violated due to order in which a request is processed > ------------------------------------------------------------------------------------------------------- > > Key: YARN-7783 > URL: https://issues.apache.org/jira/browse/YARN-7783 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Arun Suresh > Assignee: Arun Suresh > Priority: Blocker > Attachments: YARN-7783-YARN-6592.001.patch, YARN-7783-YARN-6592.002.patch, YARN-7783-YARN-6592.003.patch > > > When the algorithm has placed a container on a node, allocation tags are added to the node if the constraint is satisfied, But depending on the order in which the algorithm sees the request, it is possible that a constraint that happen to be valid during placement of an earlier-seen request, might not be valid after all subsequent requests have been placed. > For eg: > Assume nodes n1, n2, n3, n4 and n5 > Consider the 2 constraints: > # *foo* -> anti-affinity with *foo* > # *bar* -> anti-affinity with *foo* > And 2 requests > # req1: NumAllocations = 4, allocTags = [foo] > # req2: NumAllocations = 1, allocTags = [bar] > If *req1* is seen first, the algorithm can place the 4 containers in n1, n2, n3 and n4. And when it gets to *req2*, it will see that 4 nodes have the *foo* tag and will place it on n5. But if *req2* is seen first, then *bar* tag will be placed on any node, since no node will at that point have *foo*, and then when it gets to *req1*, since *foo* has no anti-affinity with *bar*, the algorithm can end up placing *foo* on a node with *bar* violating the second constraint. > To prevent the above, we need a validation step: after the placements for a batch of requests are made, then for each req, we remove its tags from the node and try to see of constraints are still satisfied if the tag were to be added back on the node. > When applied to the example above, after the algorithm has run through *req2* and then *req1*, we remove the *bar* tag from the node and try to add it back on the node. This time, constraint satisfaction will fail, since there is now a *foo* tag on the node and *bar* cannot be added. The algorithm will then retry placing *req2* on another node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org