hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ma...@apache.org
Subject svn commit: r1364928 - in /hadoop/common/branches/branch-1: CHANGES.txt bin/hadoop-config.sh
Date Tue, 24 Jul 2012 07:37:10 GMT
Author: mattf
Date: Tue Jul 24 07:37:09 2012
New Revision: 1364928

URL: http://svn.apache.org/viewvc?rev=1364928&view=rev
Log:
HADOOP-7154. Set MALLOC_ARENA_MAX in hadoop-env.sh. Contributed by Todd Lipcon.

Modified:
    hadoop/common/branches/branch-1/CHANGES.txt   (contents, props changed)
    hadoop/common/branches/branch-1/bin/hadoop-config.sh

Modified: hadoop/common/branches/branch-1/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/CHANGES.txt?rev=1364928&r1=1364927&r2=1364928&view=diff
==============================================================================
--- hadoop/common/branches/branch-1/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1/CHANGES.txt Tue Jul 24 07:37:09 2012
@@ -230,6 +230,9 @@ Release 1.1.0 - unreleased
     HDFS-2465. Add HDFS support for fadvise readahead and drop-behind.
     (todd, backported by Brandon Li via suresh)
 
+    HADOOP-7154. Should set MALLOC_ARENA_MAX in hadoop-env.sh
+    (todd via mattf)
+
     MAPREDUCE-4399. Change the Jetty response buffer size to improve 
     shuffle performance. (Luke Lu via suresh)
 

Propchange: hadoop/common/branches/branch-1/CHANGES.txt
------------------------------------------------------------------------------
  Merged /hadoop/common/branches/branch-1.1/CHANGES.txt:r1364925

Modified: hadoop/common/branches/branch-1/bin/hadoop-config.sh
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/bin/hadoop-config.sh?rev=1364928&r1=1364927&r2=1364928&view=diff
==============================================================================
--- hadoop/common/branches/branch-1/bin/hadoop-config.sh (original)
+++ hadoop/common/branches/branch-1/bin/hadoop-config.sh Tue Jul 24 07:37:09 2012
@@ -75,6 +75,11 @@ if [ "$HADOOP_HOME_WARN_SUPPRESS" = "" ]
   echo 1>&2
 fi
 
+# Newer versions of glibc use an arena memory allocator that causes virtual
+# memory usage to explode. This interacts badly with the many threads that
+# we use in Hadoop. Tune the variable down to prevent vmem explosion.
+export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-4}
+
 export HADOOP_HOME=${HADOOP_PREFIX}
 export HADOOP_HOME_WARN_SUPPRESS=1
 



Mime
View raw message