Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 43F0A6424 for ; Tue, 2 Aug 2011 04:39:47 +0000 (UTC) Received: (qmail 8941 invoked by uid 500); 2 Aug 2011 04:39:46 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 8214 invoked by uid 500); 2 Aug 2011 04:39:30 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 8154 invoked by uid 99); 2 Aug 2011 04:39:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2011 04:39:22 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.215.173] (HELO mail-ey0-f173.google.com) (209.85.215.173) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2011 04:39:17 +0000 Received: by eyb7 with SMTP id 7so8383298eyb.32 for ; Mon, 01 Aug 2011 21:38:55 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.102.67 with SMTP id f3mr1988544ebo.19.1312259933527; Mon, 01 Aug 2011 21:38:53 -0700 (PDT) Received: by 10.213.34.208 with HTTP; Mon, 1 Aug 2011 21:38:53 -0700 (PDT) In-Reply-To: References: Date: Mon, 1 Aug 2011 21:38:53 -0700 Message-ID: Subject: Re: Configuring Hadoop daemon heap sizes From: Kai Ju Liu To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=00504502d06e30df6104a97e53b8 --00504502d06e30df6104a97e53b8 Content-Type: text/plain; charset=ISO-8859-1 Hi. The new settings work perfectly for all five Hadoop daemons. Thanks so much for the example and the link! Kai Ju On Mon, Aug 1, 2011 at 11:16 AM, Harsh J wrote: > For specific max-heap sizes, you have to pass the value as a java vm > argument. See > http://avricot.com/blog/index.php?post/2010/05/03/Get-started-with-java-JVM-memory-(heap%2C-stack%2C-xss-xms-xmx-xmn.. > .) > for a good view on things with JVM and memory. > > An example for specific heap size options to JVM: > > # 1024 MB for DN > HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -Xmx1024m" > # 4 GB for NN > HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -Xmx4g" > > On Mon, Aug 1, 2011 at 11:41 PM, Kai Ju Liu wrote: > > Hi. I'm trying to tweak heap sizes for the Hadoop daemons, i.e. > > namenode/datanode and jobtracker/tasktracker. I've tried setting > > HADOOP_NAMENODE_HEAPSIZE, HADOOP_DATANODE_HEAPSIZE, and so on in > > hadoop-env.sh, but the heap size remains at the default of 1,000MB. > > > > In the cluster setup documentation, I see references to setting > > HADOOP_NAMENODE_OPTS, HADOOP_DATANODE_OPTS, and so on in hadoop-env.sh. > Is > > this the proper way to set heap sizes, and if so, what is the proper > syntax > > within the OPTS values? Thanks! > > > > Kai Ju Liu > > -- > Harsh J > --00504502d06e30df6104a97e53b8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi. The new settings work perfectly for all five Hadoop daemons. Thanks so = much for the example and the link!

Kai Ju

On Mon, Aug 1, 2011 at 11:16 AM, Harsh J <harsh@cloudera.com> wrote:<= br>
For specific max-heap sizes, you have to pa= ss the value as a java vm
argument. See http://avricot.com/blog/index.php?post/2010/05/03/Get-starte= d-with-java-JVM-memory-(heap%2C-stack%2C-xss-xms-xmx-xmn...)
for a good view on things with JVM and memory.

An example for specific heap size options to JVM:

# 1024 MB for DN
HADOOP_DATANODE_OPTS=3D"$HADOOP_DATANODE_OPTS -Xmx1024m"
# 4 GB for NN
HADOOP_NAMENODE_OPTS=3D"$HADOOP_NAMENODE_OPTS -Xmx4g"

On Mon, Aug 1, 2011 at 11:41 PM, Kai Ju Liu <kaiju@tellapart.com> wrote:
> Hi. I'm trying to tweak heap sizes for the Hadoop daemons, i.e. > namenode/datanode and jobtracker/tasktracker. I've tried setting > HADOOP_NAMENODE_HEAPSIZE, HADOOP_DATANODE_HEAPSIZE, and so on in
> hadoop-env.sh, but the heap size remains at the default of 1,000MB. >
> In the cluster setup documentation, I see references to setting
> HADOOP_NAMENODE_OPTS, HADOOP_DATANODE_OPTS, and so on in hadoop-env.sh= . Is
> this the proper way to set heap sizes, and if so, what is the proper s= yntax
> within the OPTS values? Thanks!
>
> Kai Ju Liu

--
Harsh J
--00504502d06e30df6104a97e53b8--