From yarn-issues-return-117728-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 17 14:56:04 2017 Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B721D19FF1 for ; Mon, 17 Jul 2017 14:56:04 +0000 (UTC) Received: (qmail 5036 invoked by uid 500); 17 Jul 2017 14:56:04 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 4995 invoked by uid 500); 17 Jul 2017 14:56:04 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 4984 invoked by uid 99); 17 Jul 2017 14:56:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jul 2017 14:56:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 25797C194F for ; Mon, 17 Jul 2017 14:56:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id xafnbe-Cwgkr for ; Mon, 17 Jul 2017 14:56:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 1793461AEF for ; Mon, 17 Jul 2017 14:56:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 48AFBE0931 for ; Mon, 17 Jul 2017 14:56:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 34CFA2476E for ; Mon, 17 Jul 2017 14:56:00 +0000 (UTC) Date: Mon, 17 Jul 2017 14:56:00 +0000 (UTC) From: "Nathan Roberts (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089936#comment-16089936 ] Nathan Roberts commented on YARN-6775: -------------------------------------- Attached screenshots that show a couple of before/after metrics. Change went active early on the 14th. 1) rmeventprocbusy is avg cpu busy of the Event Processor thread 2) rpceventprocessingtimeschedulerport is avg rpc processing time for the scheduler port. > CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls > -------------------------------------------------------------------------------------------------- > > Key: YARN-6775 > URL: https://issues.apache.org/jira/browse/YARN-6775 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler > Affects Versions: 2.8.1, 3.0.0-alpha3 > Reporter: Nathan Roberts > Assignee: Nathan Roberts > Fix For: 3.0.0-beta1 > > Attachments: rmeventprocbusy.png, rpcprocessingtimeschedulerport.png, YARN-6775.001.patch, YARN-6775.002.patch, YARN-6775.branch-2.002.patch, YARN-6775.branch-2.8.002.patch > > > There are several things in assignContainers() that are done multiple times even though the result cannot change (canAssignToUser, canAssignToQueue). Add some local caching to take advantage of this fact. > Will post patch shortly. Patch includes a simple throughput test that demonstrates when we have users at their user-limit, the number of NodeUpdateSchedulerEvents we can process can be improved from 13K/sec to 50K/sec. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org