hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bharat Viswanadham (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12916) HDFS commands throws error, when only shaded clients in classpath
Date Fri, 15 Dec 2017 00:32:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291840#comment-16291840
] 

Bharat Viswanadham edited comment on HDFS-12916 at 12/15/17 12:31 AM:
----------------------------------------------------------------------

[~aw] Thanks for your comments.
bq. In hadoop 3.x when HADOOP_OPTIONAL_TOOLS are loaded, they expect their dependencies from
the other directories to already be in the classpath. I'd be greatly surprised if they find
them from the shaded jar since I'm pretty sure we do class hiding.

I have included Hadoop tools lib folder, so which will be adequate to get all its dependencies.
I have just ran distcp for sample, need to run other commands to know more if they cause issue.

bq. But no matter what you do, everyone is already trained to use 'hadoop classpath'. Until
that gets changed, anything else that is done is moot.

if I override all home variables(like hdfs_home, mapred_home) to same client location in hadoop-layout.sh,
it will have only shaded client jars. so i think this way we can address hadoop classpath
problem.





was (Author: bharatviswa):
bq. In hadoop 3.x when HADOOP_OPTIONAL_TOOLS are loaded, they expect their dependencies from
the other directories to already be in the classpath. I'd be greatly surprised if they find
them from the shaded jar since I'm pretty sure we do class hiding.

I have included Hadoop tools lib folder, so which will be adequate to get all its dependencies.
I have just ran distcp for sample, need to run other commands to know more if they cause issue.

bq. But no matter what you do, everyone is already trained to use 'hadoop classpath'. Until
that gets changed, anything else that is done is moot.

if I override all home variables(like hdfs_home, mapred_home) to same client location in hadoop-layout.sh,
it will have only shaded client jars. so i think this way we can address hadoop classpath
problem.




> HDFS commands throws error, when only shaded clients in classpath
> -----------------------------------------------------------------
>
>                 Key: HDFS-12916
>                 URL: https://issues.apache.org/jira/browse/HDFS-12916
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Bharat Viswanadham
>            Assignee: Bharat Viswanadham
>
> [root@n001 hadoop]# bin/hdfs dfs -rm /
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
> 	at org.apache.hadoop.fs.FsShell.run(FsShell.java:303)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> 	at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.ClassNotFoundException: org.apache.htrace.core.Tracer$Builder
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 	... 4 more
> cc [~busbey]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message