hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "tang shanjiang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5643) DynamicMR: A Dynamic Slot Utilization Optimization Framework for Hadoop MRv1
Date Fri, 22 Nov 2013 09:16:36 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

tang shanjiang updated MAPREDUCE-5643:
--------------------------------------

    Status: Patch Available  (was: Open)

> DynamicMR: A Dynamic Slot Utilization Optimization Framework for Hadoop MRv1
> ----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5643
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5643
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 1.2.1
>            Reporter: tang shanjiang
>            Assignee: tang shanjiang
>              Labels: performance
>
> Hadoop MRv1 uses the slot-based resource model with the static configuration of map/reduce
slots in advance. Due to the rigid execution order between map and reduce tasks in a MapReduce
environment and the strict execution constrain that map tasks can only run map slots and reduce
tasks can only reduce slots, slots can be severely under-utilized, which significantly degrades
the performance. 
> In contrast to YARN that gives up the slot-based resource model to maximize resource
utilization, we keep the slot-based model and propose a dynamic slot utilization optimization
system called DynamicMR to improve the performance of Hadoop by maximizing the slots utilization
and improving utilization efficiency while guaranteeing the fairness across pools. It consists
of three levels of scheduling components, namely, Dynamic Hadoop Fair Scheduler (DHFS), Dynamic
Speculative Task Scheduler (DSTS), and Data Locality Maximization Scheduler (DLMS).
> Our tests show that DynamicMR outperforms YARN for MapReduce workloads with multiple
jobs, especially when the number of jobs is large.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message