Return-Path: X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 49B411004A for ; Fri, 22 Nov 2013 09:13:03 +0000 (UTC) Received: (qmail 91513 invoked by uid 500); 22 Nov 2013 09:12:57 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 91364 invoked by uid 500); 22 Nov 2013 09:12:37 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 91223 invoked by uid 99); 22 Nov 2013 09:12:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Nov 2013 09:12:35 +0000 Date: Fri, 22 Nov 2013 09:12:35 +0000 (UTC) From: "tang shanjiang (JIRA)" To: mapreduce-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (MAPREDUCE-5643) DynamicMR: A Dynamic Slot Utilization Optimization Framework for Hadoop MRv1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 tang shanjiang created MAPREDUCE-5643: ----------------------------------------- Summary: DynamicMR: A Dynamic Slot Utilization Optimization Framework for Hadoop MRv1 Key: MAPREDUCE-5643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5643 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Affects Versions: 1.2.1 Reporter: tang shanjiang Hadoop MRv1 uses the slot-based resource model with the static configuration of map/reduce slots in advance. Due to the rigid execution order between map and reduce tasks in a MapReduce environment and the strict execution constrain that map tasks can only run map slots and reduce tasks can only reduce slots, slots can be severely under-utilized, which significantly degrades the performance. In contrast to YARN that gives up the slot-based resource model to maximize resource utilization, we keep the slot-based model and propose a dynamic slot utilization optimization system called DynamicMR to improve the performance of Hadoop by maximizing the slots utilization and improving utilization efficiency while guaranteeing the fairness across pools. It consists of three levels of scheduling components, namely, Dynamic Hadoop Fair Scheduler (DHFS), Dynamic Speculative Task Scheduler (DSTS), and Data Locality Maximization Scheduler (DLMS). Our tests show that DynamicMR outperforms YARN for MapReduce workloads with multiple jobs, especially when the number of jobs is large. -- This message was sent by Atlassian JIRA (v6.1#6144)