Return-Path: X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C7DE11050D for ; Mon, 25 Nov 2013 05:32:50 +0000 (UTC) Received: (qmail 64405 invoked by uid 500); 25 Nov 2013 05:32:43 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 64336 invoked by uid 500); 25 Nov 2013 05:32:37 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 64319 invoked by uid 99); 25 Nov 2013 05:32:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Nov 2013 05:32:35 +0000 Date: Mon, 25 Nov 2013 05:32:35 +0000 (UTC) From: "stanley shi (JIRA)" To: mapreduce-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 stanley shi created MAPREDUCE-5649: -------------------------------------- Summary: Reduce cannot use more than 2G memory for the final merge Key: MAPREDUCE-5649 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: stanley shi In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method: int maxInMemReduce = (int)Math.min( Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE); This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts. -- This message was sent by Atlassian JIRA (v6.1#6144)