Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 33C3C112A3 for ; Thu, 28 Aug 2014 15:38:09 +0000 (UTC) Received: (qmail 6203 invoked by uid 500); 28 Aug 2014 15:38:08 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 6156 invoked by uid 500); 28 Aug 2014 15:38:08 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 6142 invoked by uid 99); 28 Aug 2014 15:38:08 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Aug 2014 15:38:08 +0000 Date: Thu, 28 Aug 2014 15:38:08 +0000 (UTC) From: "Allen Wittenauer (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-11013) CLASSPATH handling should be consolidated, debuggable MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113863#comment-14113863 ] Allen Wittenauer commented on HADOOP-11013: ------------------------------------------- Some examples: Namenode: {code} $ bin/hdfs --debug --daemon start namenode DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/* DEBUG: Appending HADOOP_NAMENODE_OPTS onto HADOOP_OPTS DEBUG: HADOOP_OPTS declined -Xmx1024m DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender WARNING: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs does not exist. Creating. {code} ResourceManager: {code} $ bin/yarn --debug --daemon start resourcemanager DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/* DEBUG: Append YARN_RESOURCEMANAGER_OPTS onto YARN_OPTS DEBUG: Resetting HADOOP_OPTS=YARN_OPTS DEBUG: HADOOP_OPTS accepted -Xmx1024m DEBUG: HADOOP_OPTS accepted -Dyarn.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dyarn.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log DEBUG: HADOOP_OPTS accepted -Dyarn.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS declined -Dyarn.root.logger=INFO,RFA DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw DEBUG: HADOOP_OPTS declined -Dhadoop.root.logger=INFO,RFA DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender {code} distcp {code} $ bin/hadoop --debug distcp /tmp /1 DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/* DEBUG: Injecting TOOL_PATH into CLASSPATH DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/lib/* DEBUG: Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS DEBUG: HADOOP_OPTS accepted -Xmx1024m DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop.log DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,console DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender DEBUG: Final CLASSPATH: /Users/aw/HADOOP/conf:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/lib/* DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc= -Djava.security.krb5.conf= -Xmx1024m -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT -Dhadoop.id.str=aw -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender {code} > CLASSPATH handling should be consolidated, debuggable > ----------------------------------------------------- > > Key: HADOOP-11013 > URL: https://issues.apache.org/jira/browse/HADOOP-11013 > Project: Hadoop Common > Issue Type: Improvement > Reporter: Allen Wittenauer > Assignee: Allen Wittenauer > Attachments: HADOOP-11013-01.patch, HADOOP-11013.patch > > > As part of HADOOP-9902, java execution across many different shell bits were consolidated down to (effectively) two routines. Prior to calling those two routines, the CLASSPATH is exported. This export should really be getting handled in the exec function and not in the individual shell bits. > Additionally, it would be good if there was: > {code} > echo ${CLASSPATH} > /dev/null > {code} > so that bash -x would show the content of the classpath or even a '--debug classpath' option that would echo the classpath to the screen prior to java exec to help with debugging. -- This message was sent by Atlassian JIRA (v6.2#6252)