hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11013) CLASSPATH handling should be consolidated, debuggable
Date Thu, 28 Aug 2014 15:38:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113863#comment-14113863
] 

Allen Wittenauer commented on HADOOP-11013:
-------------------------------------------

Some examples:

Namenode: 

{code}
$ bin/hdfs --debug --daemon start namenode
DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
DEBUG: Appending HADOOP_NAMENODE_OPTS onto HADOOP_OPTS
DEBUG: HADOOP_OPTS declined -Xmx1024m
DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log
DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA
DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender
WARNING: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs does not exist. Creating.
{code}

ResourceManager:
{code}
$ bin/yarn --debug --daemon start resourcemanager
DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
DEBUG: Append YARN_RESOURCEMANAGER_OPTS onto YARN_OPTS
DEBUG: Resetting HADOOP_OPTS=YARN_OPTS
DEBUG: HADOOP_OPTS accepted -Xmx1024m
DEBUG: HADOOP_OPTS accepted -Dyarn.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
DEBUG: HADOOP_OPTS accepted -Dyarn.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log
DEBUG: HADOOP_OPTS accepted -Dyarn.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
DEBUG: HADOOP_OPTS declined -Dyarn.root.logger=INFO,RFA
DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log
DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
DEBUG: HADOOP_OPTS declined -Dhadoop.root.logger=INFO,RFA
DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender
{code}

distcp
{code}
$ bin/hadoop --debug distcp /tmp /1
DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
DEBUG: Injecting TOOL_PATH into CLASSPATH
DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/lib/*
DEBUG: Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS
DEBUG: HADOOP_OPTS accepted -Xmx1024m
DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop.log
DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,console
DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender
DEBUG: Final CLASSPATH: /Users/aw/HADOOP/conf:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/lib/*
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm=  -Djava.security.krb5.kdc=
 -Djava.security.krb5.conf=   -Xmx1024m -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
-Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT -Dhadoop.id.str=aw
-Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
{code}

> CLASSPATH handling should be consolidated, debuggable
> -----------------------------------------------------
>
>                 Key: HADOOP-11013
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11013
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>         Attachments: HADOOP-11013-01.patch, HADOOP-11013.patch
>
>
> As part of HADOOP-9902, java execution across many different shell bits were consolidated
down to (effectively) two routines.  Prior to calling those two routines, the CLASSPATH is
exported.  This export should really be getting handled in the exec function and not in the
individual shell bits.
> Additionally, it would be good if there was:
> {code}
> echo ${CLASSPATH} > /dev/null
> {code}
> so that bash -x would show the content of the classpath or even a '--debug classpath'
option that would echo the classpath to the screen prior to java exec to help with debugging.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message