From mapreduce-dev-return-8492-apmail-hadoop-mapreduce-dev-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 27 14:35:47 2012 Return-Path: X-Original-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6A039D807 for ; Sat, 27 Oct 2012 14:35:47 +0000 (UTC) Received: (qmail 86161 invoked by uid 500); 27 Oct 2012 14:35:46 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 85973 invoked by uid 500); 27 Oct 2012 14:35:46 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 85947 invoked by uid 99); 27 Oct 2012 14:35:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Oct 2012 14:35:45 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of csxulijie@gmail.com designates 209.85.220.48 as permitted sender) Received: from [209.85.220.48] (HELO mail-pa0-f48.google.com) (209.85.220.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Oct 2012 14:35:35 +0000 Received: by mail-pa0-f48.google.com with SMTP id kp12so2562420pab.35 for ; Sat, 27 Oct 2012 07:35:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type; bh=py73LlRUieDAD2vedMIrmeUerXWHHdCpop6LusP3gU8=; b=MPms8fe6UeeRsh2NDdJ2x7Kt+rGBik1EF0O9g5hya6uQs/X490APxdO9ucfJF3cyqd MZic4CbeXOKuMlK9blxq0mGM2FfMKquasNShc04HJu+nDzyDZjgVs6Ms3MUexPgmBuLo q6kyjtGySoKW1n7jqvbcxzaD+/0dPYRdaPiZefM35jRBqJhL/4sirrVaU8Jy5PEa4GuW vpN5JZXviaSzMRGn59n4PIh98d+7TaHaV7u0gCB+kJypUuOF0MLfEfOGehR/Ek+RzV2Q RyuGpr7jfIWU4EI3c8MY/SEIDcaQvZoLSdEmcP0U4naDBvEM80ljRQ0xLatrv94alNfH YjDA== Received: by 10.68.241.133 with SMTP id wi5mr78425814pbc.48.1351348514402; Sat, 27 Oct 2012 07:35:14 -0700 (PDT) Received: from [133.133.134.19] ([124.16.137.194]) by mx.google.com with ESMTPS id wf8sm2886391pbc.65.2012.10.27.07.35.12 (version=SSLv3 cipher=OTHER); Sat, 27 Oct 2012 07:35:13 -0700 (PDT) Message-ID: <508BF11B.4090909@gmail.com> Date: Sat, 27 Oct 2012 22:35:07 +0800 From: Lijie Xu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: mapreduce-dev@hadoop.apache.org Subject: About reducer's Shuffle JVM Heap Size Content-Type: multipart/alternative; boundary="------------030603040705030902020604" X-Virus-Checked: Checked by ClamAV on apache.org --------------030603040705030902020604 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, all. I'm debugging Hadoop's source code and find an incomprehensible setting. In hadoop-0.20.2 and hadoop-1.0.3, reducer's shuffle buffer size cannot exceed 2048MB (i.e., Integer.MAX_VALUE). In this way, although*//* reducer's JVM size can be set more than 2048MB (e.g., mapred.child.java.opts=-Xmx4000m), the heap size used for shuffle buffer is at most "2048MB * maxInMemCopyUse (default 0.7)" not "4000MB * maxInMemCopyUse". I think it's not a reasonable setting for large memory machines. The following code taken from "org.apache.hadoop.mapred.ReduceTask" shows the concrete algorithm. I'm wondering why maxSize is declared as "long" but set as "(int)". I point out the corresponding code with "-->" Thanks for any suggestion. --------------------------------------------------------------------------------------------------------------------------- private final long maxSize; private final long maxSingleShuffleLimit; private long size = 0; private Object dataAvailable = new Object(); private long fullSize = 0; private int numPendingRequests = 0; private int numRequiredMapOutputs = 0; private int numClosed = 0; private boolean closed = false; public ShuffleRamManager(Configuration conf) throws IOException { final float maxInMemCopyUse = conf.getFloat("mapred.job.shuffle.input.buffer.percent", 0.70f); if (maxInMemCopyUse > 1.0 || maxInMemCopyUse < 0.0) { throw new IOException("mapred.job.shuffle.input.buffer.percent" + maxInMemCopyUse); } // Allow unit tests to fix Runtime memory --> maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes", --> (int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE)) --> * maxInMemCopyUse); maxSingleShuffleLimit = (long)(maxSize * MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION); LOG.info("ShuffleRamManager: MemoryLimit=" + maxSize + ", MaxSingleShuffleLimit=" + maxSingleShuffleLimit); } --------------030603040705030902020604--