hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-11013) CLASSPATH handling should be consolidated, debuggable
Date Thu, 28 Aug 2014 03:27:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113262#comment-14113262
] 

Allen Wittenauer edited comment on HADOOP-11013 at 8/28/14 3:27 AM:
--------------------------------------------------------------------

Some sample output:

{code}
$ bin/hdfs --debug --daemon start namenode
DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
DEBUG: HADOOP_OPTS declined -Xmx1024m
DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log
DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA
DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender

aw-mbp-work:hadoop-3.0.0-SNAPSHOT aw$ more logs/*out
DEBUG: Final CLASSPATH: /Users/aw/HADOOP/conf:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm=  -Djava.security.krb5.kdc=
 -Djava.security.krb5.conf=  -Xmx1g -XX:HeapDumpPath=./java_pid_<pid>.hprof -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
-Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
-Dhadoop.id.str=aw -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
{code}


was (Author: aw):
Some sample output:

{code}
	

    $ bin/hdfs --debug --daemon start namenode
    DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
    DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
    DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
    DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
    DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
    DEBUG: HADOOP_OPTS declined -Xmx1024m
    DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
    DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
    DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log
    DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
    DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
    DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA
    DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
    DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender
    aw-mbp-work:hadoop-3.0.0-SNAPSHOT aw$ more logs/*out
    DEBUG: Final CLASSPATH: /Users/aw/HADOOP/conf:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
    DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm=
 -Djava.security.krb5.kdc=  -Djava.security.krb5.conf=  -Xmx1g -XX:HeapDumpPath=./java_pid_<pid>.hprof
-Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log
-Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT -Dhadoop.id.str=aw -Dhadoop.root.logger=INFO,RFA
-Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
{code}



> CLASSPATH handling should be consolidated, debuggable
> -----------------------------------------------------
>
>                 Key: HADOOP-11013
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11013
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>         Attachments: HADOOP-11013.patch
>
>
> As part of HADOOP-9902, java execution across many different shell bits were consolidated
down to (effectively) two routines.  Prior to calling those two routines, the CLASSPATH is
exported.  This export should really be getting handled in the exec function and not in the
individual shell bits.
> Additionally, it would be good if there was:
> {code}
> echo ${CLASSPATH} > /dev/null
> {code}
> so that bash -x would show the content of the classpath or even a '--debug classpath'
option that would echo the classpath to the screen prior to java exec to help with debugging.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message