hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Charles (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7154) Should set MALLOC_ARENA_MAX in hadoop-config.sh
Date Wed, 25 Jul 2012 12:47:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422206#comment-13422206
] 

Eric Charles commented on HADOOP-7154:
--------------------------------------

I wonder if MALLOC_ARENA_MAX="4" fixes everything
More test on my ubuntu12 laptop dev env (with export MALLOC_ARENA_MAX=4 in .bashrc):

[1] (Pi Estimator) and [2] (RandomWriter) complete successfully.

[3] DistributedShell (see 'is running beyond virtual memory limits' message):

12/07/25 14:23:15 INFO distributedshell.Client: Got application report from ASM for, appId=3,
clientToken=null, appDiagnostics=Application application_1343218758332_0003 failed 1 times
due to AM Container for appattempt_1343218758332_0003_000001 exited with  exitCode: 143 due
to: Container [pid=31510,containerID=container_1343218758332_0003_01_000001] is running beyond
virtual memory limits. Current usage: 82.8mb of 128.0mb physical memory used; 873.6mb of 268.8mb
virtual memory used. Killing container.
Dump of the process-tree for container_1343218758332_0003_01_000001 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 31510 28157 31510 31510 (bash) 0 1 17031168 369 /bin/bash -c /d/opt/jdk1.6.0_31/bin/java
-Xmx128m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory
10 --num_containers 1 --priority 0 --shell_command ls --shell_args / 1>/data/hadoop-3.0.0-SNAPSHOT-yarn-log-dirs/application_1343218758332_0003/container_1343218758332_0003_01_000001/AppMaster.stdout
2>/data/hadoop-3.0.0-SNAPSHOT-yarn-log-dirs/application_1343218758332_0003/container_1343218758332_0003_01_000001/AppMaster.stderr
  
	|- 31514 31510 31510 31510 (java) 139 10 899014656 20830 /d/opt/jdk1.6.0_31/bin/java -Xmx128m
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory
10 --num_containers 1 --priority 0 --shell_command ls --shell_args / 
.Failing this attempt.. Failing the application., appMasterHost=, appQueue=default, appMasterRpcPort=0,
appStartTime=1343218992022, yarnAppState=FAILED, distributedFinalState=FAILED, appTrackingUrl=,
appUser=echarles


Looking at the Yarn tmp files, I see that MALLOC_ARENA_MAX=4 is effectively exported:
more ./usercache/echarles/appcache/application_1343218758332_0003/container_1343218758332_0003_01_000001/launch_container.sh
| grep MALLOC
export MALLOC_ARENA_MAX="4"



Thx, Eric


[1] hadoop jar ${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 100
100
[2] hadoop jar ${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar randomwriter
randomwriter-out
[3] hadoop jar ${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-yarn-applications-distributedshell-*.jar
org.apache.hadoop.yarn.applications.distributedshell.Client -jar ${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar
-shell_command pwd -num_containers 1

                
> Should set MALLOC_ARENA_MAX in hadoop-config.sh
> -----------------------------------------------
>
>                 Key: HADOOP-7154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 1.1.0, 0.22.0
>
>         Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design. In several
clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since
when multiple threads perform allocations, they each get their own memory arena. On a 64-bit
system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number
of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This
causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and
bound the virtual memory, with no noticeable downside in performance - we've been recommending
MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes
more and more common.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message