hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Grandl (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2745) Extend YARN to support multi-resource packing of tasks
Date Mon, 15 Dec 2014 23:38:15 GMT

     [ https://issues.apache.org/jira/browse/YARN-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Grandl updated YARN-2745:
--------------------------------
    Component/s: nodemanager
    Description: In this umbrella JIRA we propose an extension to existing scheduling techniques,
which accounts for all resources used by a task (CPU, memory, disk, network) and it is able
to achieve three competing objectives: fairness, improve cluster utilization and reduces average
job completion time.  (was: In this umbrella JIRA we propose a new pluggable scheduler, which
accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve
three competing objectives: fairness, improve cluster utilization and reduces average job
completion time.)
        Summary: Extend YARN to support multi-resource packing of tasks  (was: YARN new pluggable
scheduler which does multi-resource packing)

Summary of main changes
* Update the container allocation logic in the RM scheduler. This change is the core. It enables
“packing” tasks, preferring jobs with less remaining work and trades off fairness for
efficiency. 

* Expand the AM->RM resource ask to expose tasks’ disk and network resource demands to
the scheduler. 

*Support for cluster-wide resource tracking: want per-machine resource usage information available
at the RM.  


> Extend YARN to support multi-resource packing of tasks
> ------------------------------------------------------
>
>                 Key: YARN-2745
>                 URL: https://issues.apache.org/jira/browse/YARN-2745
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager, scheduler
>            Reporter: Robert Grandl
>         Attachments: sigcomm_14_tetris_talk.pptx, tetris_paper.pdf
>
>
> In this umbrella JIRA we propose an extension to existing scheduling techniques, which
accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve
three competing objectives: fairness, improve cluster utilization and reduces average job
completion time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message