hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5164) CapacityOvertimePolicy does not take advantaged of plan RLE
Date Fri, 27 May 2016 21:40:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304849#comment-15304849

Carlo Curino commented on YARN-5164:

Thinking more about this, it is non-trivial to RLE-ify this policy. The policy computes for
every sliding window the integral of used resources and make sure it does not exceed an integral
quota. The issue with this is that even for a constant value of the user reservation (or some
of reservations) the computed integral over a moving window changes.
In the attached *YARN-5164-example.pdf* we visualize this for a simple 5-sample window. The
orange line changes even for constant values of the blue line. 

We could in principle leverage higher-order representations (where we store RLE-encoded vectors
for non-horizontal lines as starting-point + slope), but this is far from trivial, and we
need to make sure the code complexity is justified. Postponing this for now. 

> CapacityOvertimePolicy does not take advantaged of plan RLE
> -----------------------------------------------------------
>                 Key: YARN-5164
>                 URL: https://issues.apache.org/jira/browse/YARN-5164
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler, fairscheduler, resourcemanager
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-5164-example.pdf
> As a consequence small time granularities (e.g., 1 sec) and long time horizon for a reservation
(e.g., months) run rather slow (10 sec). 
> Proposed resolution is to switch to interval math in checking, similar to how YARN-4359
does for agents.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message