hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ishai Menache (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3656) LowCost: A Cost-Based Placement Agent for YARN Reservations
Date Tue, 14 Jul 2015 21:35:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627082#comment-14627082

Ishai Menache commented on YARN-3656:

Thank you [~asuresh] for your comments!
We will remove the redundant “if” conditions from TryManyReservationAgents.java.

Regarding your suggestion of incorporating a fallback strategy for ReservationAgent – it
is an interesting idea, but we believe it might lead to a couple of issues.
The behavior of TryManyReservationAgents is not straightforward. For example, if you have
an agent defined by RA = TryManyReservationAgents{ RA1, RA2, RA3 }, the overall performance
of RA depends on how the algorithms RA1, RA2 and RA3 affect one another, and this dependency
can be complex. In general, you might want to implement several TryMany policies with different
fallback policies, depending on the available capacity, job characteristics etc. This is why
we preferred to implement the collection of the algorithms as a LinkedList, which gives full
flexibility to the user. On a related note, the fallback setting might accidentally end up
in a cyclic dependency between the different algorithms. Please let us know if you agree.

> LowCost: A Cost-Based Placement Agent for YARN Reservations
> -----------------------------------------------------------
>                 Key: YARN-3656
>                 URL: https://issues.apache.org/jira/browse/YARN-3656
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler, resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Ishai Menache
>            Assignee: Jonathan Yaniv
>              Labels: capacity-scheduler, resourcemanager
>         Attachments: LowCostRayonExternal.pdf, YARN-3656-v1.1.patch, YARN-3656-v1.2.patch,
YARN-3656-v1.patch, lowcostrayonexternal_v2.pdf
> YARN-1051 enables SLA support by allowing users to reserve cluster capacity ahead of
time. YARN-1710 introduced a greedy agent for placing user reservations. The greedy agent
makes fast placement decisions but at the cost of ignoring the cluster committed resources,
which might result in blocking the cluster resources for certain periods of time, and in turn
rejecting some arriving jobs.
> We propose LowCost – a new cost-based planning algorithm. LowCost “spreads” the
demand of the job throughout the allowed time-window according to a global, load-based cost

This message was sent by Atlassian JIRA

View raw message