hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3656) LowCost: A Cost-Based Placement Agent for YARN Reservations
Date Wed, 15 Jul 2015 07:40:06 GMT

    [ https://issues.apache.org/jira/browse/YARN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627669#comment-14627669

Arun Suresh commented on YARN-3656:

[~imenache], Yup.. makes sense
I guess a possible future improvement (probably beyond the scope of this JIRA) would be to
allow declarative configuration of the Planner (via some xml etc. to allow users to string
together different PAs). 

With respect to the algorithm and its implementation, it looks good to me.
Minor observation:
in {{StageAllocatorLowCostAligned}}, if you extract out how you are creating your {{durationIntervalsSortedByCost}}
set (lines 93 - 122) into a separate function. you can probably have different implementations
of {{StageAllocatorLowCost}} (Exhastive / Sample / Aligned) and plug it in via some configuration.

+1 otherwise

> LowCost: A Cost-Based Placement Agent for YARN Reservations
> -----------------------------------------------------------
>                 Key: YARN-3656
>                 URL: https://issues.apache.org/jira/browse/YARN-3656
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler, resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Ishai Menache
>            Assignee: Jonathan Yaniv
>              Labels: capacity-scheduler, resourcemanager
>         Attachments: LowCostRayonExternal.pdf, YARN-3656-v1.1.patch, YARN-3656-v1.2.patch,
YARN-3656-v1.patch, lowcostrayonexternal_v2.pdf
> YARN-1051 enables SLA support by allowing users to reserve cluster capacity ahead of
time. YARN-1710 introduced a greedy agent for placing user reservations. The greedy agent
makes fast placement decisions but at the cost of ignoring the cluster committed resources,
which might result in blocking the cluster resources for certain periods of time, and in turn
rejecting some arriving jobs.
> We propose LowCost – a new cost-based planning algorithm. LowCost “spreads” the
demand of the job throughout the allowed time-window according to a global, load-based cost

This message was sent by Atlassian JIRA

View raw message