hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Yan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10245) Hadoop command line always appends "-Xmx" option twice
Date Tue, 21 Jan 2014 20:45:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877837#comment-13877837

Wei Yan commented on HADOOP-10245:

[~shanyu], as discussed, there are multiple places configuring -Xmx. In the lastest patch
in HADOOP-9870 provided by [~jhsenjaliya], $HADOOP_HEAPSIZE is checked firstly; if not set,
assign -Xmx512m. Additionally, in bin/hadoop, also check the -Xmx configuration, to avoid
duplicate configurations.
Simply remove -Xmx512m from HADOOP_CLIENT_OPTS may still generate multiple -Xmx, as bin/hadoop
also has a default $JAVA_HEAP_MAX, which is 1000m.
IMO, I think HADOOP-9870 has fixed this issue.

> Hadoop command line always appends "-Xmx" option twice
> ------------------------------------------------------
>                 Key: HADOOP-10245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10245
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: bin
>    Affects Versions: 2.2.0
>            Reporter: shanyu zhao
>            Assignee: shanyu zhao
>         Attachments: HADOOP-10245.patch
> The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java with "-Xmx"
options twice. The impact is that any user defined HADOOP_HEAP_SIZE env variable will take
no effect because it is overwritten by the second "-Xmx" option.
> For example, here is the java cmd generated for command "hadoop fs -ls /", Notice that
there are two "-Xmx" options: "-Xmx1000m" and "-Xmx512m" in the command line:
> java -Xmx1000m  -Dhadoop.log.dir=C:\tmp\logs -Dhadoop.log.file=hadoop.log -Dhadoop.root.logger=INFO,c
> onsole,DRFA -Xmx512m  -Dhadoop.security.logger=INFO,RFAS -classpath XXX org.apache.hadoop.fs.FsShell
-ls /
> Here is the root cause:
> The call flow is: hadoop.sh calls hadoop_config.sh, which in turn calls hadoop-env.sh.

> In hadoop.sh, the command line is generated by the following pseudo code:
> java $JAVA_HEAP_MAX $HADOOP_CLIENT_OPTS -classpath ...
> In hadoop-config.sh, $JAVA_HEAP_MAX is initialized as "-Xmx1000m" if user didn't set
$HADOOP_HEAP_SIZE env variable.
> In hadoop-env.sh, $HADOOP_CLIENT_OPTS is set as this:
> To fix this problem, we should remove the "-Xmx512m" from HADOOP_CLIENT_OPTS. If we really
want to change the memory settings we need to use $HADOOP_HEAP_SIZE env variable.

This message was sent by Atlassian JIRA

View raw message