Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1DD36105DF for ; Tue, 21 Jan 2014 20:45:23 +0000 (UTC) Received: (qmail 1928 invoked by uid 500); 21 Jan 2014 20:45:21 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 1899 invoked by uid 500); 21 Jan 2014 20:45:20 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 1888 invoked by uid 99); 21 Jan 2014 20:45:20 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jan 2014 20:45:20 +0000 Date: Tue, 21 Jan 2014 20:45:20 +0000 (UTC) From: "Wei Yan (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-10245) Hadoop command line always appends "-Xmx" option twice MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-10245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877837#comment-13877837 ] Wei Yan commented on HADOOP-10245: ---------------------------------- [~shanyu], as discussed, there are multiple places configuring -Xmx. In the lastest patch in HADOOP-9870 provided by [~jhsenjaliya], $HADOOP_HEAPSIZE is checked firstly; if not set, assign -Xmx512m. Additionally, in bin/hadoop, also check the -Xmx configuration, to avoid duplicate configurations. Simply remove -Xmx512m from HADOOP_CLIENT_OPTS may still generate multiple -Xmx, as bin/hadoop also has a default $JAVA_HEAP_MAX, which is 1000m. IMO, I think HADOOP-9870 has fixed this issue. > Hadoop command line always appends "-Xmx" option twice > ------------------------------------------------------ > > Key: HADOOP-10245 > URL: https://issues.apache.org/jira/browse/HADOOP-10245 > Project: Hadoop Common > Issue Type: Bug > Components: bin > Affects Versions: 2.2.0 > Reporter: shanyu zhao > Assignee: shanyu zhao > Attachments: HADOOP-10245.patch > > > The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java with "-Xmx" options twice. The impact is that any user defined HADOOP_HEAP_SIZE env variable will take no effect because it is overwritten by the second "-Xmx" option. > For example, here is the java cmd generated for command "hadoop fs -ls /", Notice that there are two "-Xmx" options: "-Xmx1000m" and "-Xmx512m" in the command line: > java -Xmx1000m -Dhadoop.log.dir=C:\tmp\logs -Dhadoop.log.file=hadoop.log -Dhadoop.root.logger=INFO,c > onsole,DRFA -Xmx512m -Dhadoop.security.logger=INFO,RFAS -classpath XXX org.apache.hadoop.fs.FsShell -ls / > Here is the root cause: > The call flow is: hadoop.sh calls hadoop_config.sh, which in turn calls hadoop-env.sh. > In hadoop.sh, the command line is generated by the following pseudo code: > java $JAVA_HEAP_MAX $HADOOP_CLIENT_OPTS -classpath ... > In hadoop-config.sh, $JAVA_HEAP_MAX is initialized as "-Xmx1000m" if user didn't set $HADOOP_HEAP_SIZE env variable. > In hadoop-env.sh, $HADOOP_CLIENT_OPTS is set as this: > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > To fix this problem, we should remove the "-Xmx512m" from HADOOP_CLIENT_OPTS. If we really want to change the memory settings we need to use $HADOOP_HEAP_SIZE env variable. -- This message was sent by Atlassian JIRA (v6.1.5#6160)