Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 27 May 2016 21:40:12 +0000 (UTC)
From: "Carlo Curino (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12973210.1464300200000.321309.1464385212955@Atlassian.JIRA>
In-Reply-To: <JIRA.12973210.1464300200000@Atlassian.JIRA>
References: <JIRA.12973210.1464300200000@Atlassian.JIRA> <JIRA.12973210.1464300200284@arcas>
Subject: [jira] [Commented] (YARN-5164) CapacityOvertimePolicy does not take
 advantaged of plan RLE
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 27 May 2016 21:40:14 -0000


    [ https://issues.apache.org/jira/browse/YARN-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304849#comment-15304849 ] 

Carlo Curino commented on YARN-5164:
------------------------------------

Thinking more about this, it is non-trivial to RLE-ify this policy. The policy computes for every sliding window the integral of used resources and make sure it does not exceed an integral quota. The issue with this is that even for a constant value of the user reservation (or some of reservations) the computed integral over a moving window changes.
In the attached *YARN-5164-example.pdf* we visualize this for a simple 5-sample window. The orange line changes even for constant values of the blue line. 

We could in principle leverage higher-order representations (where we store RLE-encoded vectors for non-horizontal lines as starting-point + slope), but this is far from trivial, and we need to make sure the code complexity is justified. Postponing this for now. 

> CapacityOvertimePolicy does not take advantaged of plan RLE
> -----------------------------------------------------------
>
>                 Key: YARN-5164
>                 URL: https://issues.apache.org/jira/browse/YARN-5164
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler, fairscheduler, resourcemanager
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-5164-example.pdf
>
>
> As a consequence small time granularities (e.g., 1 sec) and long time horizon for a reservation (e.g., months) run rather slow (10 sec). 
> Proposed resolution is to switch to interval math in checking, similar to how YARN-4359 does for agents.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org