hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Akira Ajisaka (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5643) DynamicMR: A Dynamic Slot Utilization Optimization Framework for Hadoop MRv1
Date Sun, 04 Mar 2018 15:42:00 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Akira Ajisaka updated MAPREDUCE-5643:
    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

branch-1 is EoL. Closing this.

> DynamicMR: A Dynamic Slot Utilization Optimization Framework for Hadoop MRv1
> ----------------------------------------------------------------------------
>                 Key: MAPREDUCE-5643
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5643
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 1.2.1
>            Reporter: tang shanjiang
>            Assignee: tang shanjiang
>            Priority: Major
>              Labels: BB2015-05-TBR, performance
>         Attachments: DynamicMR A Dynamic Slot Allocation Optimization Framework for MapReduce
Clusters.pdf, DynamicMR-0.1.1-patch, DynamicMR_TCC_SupplementalMaterial.pdf, README
> Hadoop MRv1 uses the slot-based resource model with the static configuration of map/reduce
slots. There is a strict utility constrain that map tasks can only run on map slots and reduce
tasks can only use reduce slots. Due to the rigid execution order between map and reduce tasks
in a MapReduce environment, slots can be severely under-utilized, which significantly degrades
the performance. 
> In contrast to YARN that gives up the slot-based resource model and propose a container-based
model to maximize the resource utilization via unawareness of the types of map/reduce tasks,
we keep the slot-based model and propose a dynamic slot utilization optimization system called
DynamicMR to improve the performance of Hadoop by maximizing the slots utilization as well
as slot utilization efficiency while guaranteeing the fairness across pools. It consists of
three types of scheduling components, namely, Dynamic Hadoop Fair Scheduler (DHFS), Dynamic
Speculative Task Scheduler (DSTS), and Data Locality Maximization Scheduler (DLMS).
> Our tests show that DynamicMR outperforms YARN for MapReduce workloads with multiple
jobs, especially when the number of jobs is large. The explanation is that, given a certain
number of resources, it is obvious that the performance for the case with a ratio control
of concurrently running map and reduce tasks is better than without control. Because without
control, it easily occurs that there are too many reduce tasks running, causing the network
to be a bottleneck seriously. For YARN, both map and reduce tasks can run on any idle container.
There is no control mechanism for the ratio of resource allocation between map and reduce
tasks. It means that when there are pending reduce tasks, the idle container will be most
likely possessed by them. In contrast, DynamicMR follows the traditional slot-based model.
In contrast to the ’hard’ constrain of slot allocation that map slots have to be allocated
to map tasks and reduce tasks should be dispatched to reduce tasks, DynamicMR obeys a ’soft’
constrain of slot allocation to allow that map slot can be allocated to reduce task and vice
versa. But whenever there are pending map tasks, the map slot should be given to map tasks
first, and the rule is similar for reduce tasks. It means that, the traditional way of static
map/reduce slot configuration for the ratio control of running map/reduce tasks still works
for DynamicMR. In comparison to YARN which maximizes the resource utilization only, DynamicMR
can maximize the slot resource utilization and meanwhile dynamically control the ratio of
running map/reduce tasks via map/reduce slot configuration.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message