hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3845) hadoop distcp fails to run with java.lang.NoClassDefFoundError
Date Thu, 09 Feb 2012 23:47:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205055#comment-13205055
] 

Jason Lowe commented on MAPREDUCE-3845:
---------------------------------------

This seems related to the same type of problem as in HADOOP-7999 which should be fixed in
0.23.  I'm unable to reproduce the problem, as I get the same output from both commands:

{noformat}
$ ls
hadoop-0.23.1.tar.gz
$ tar xf hadoop\-0.23.1.tar.gz 
$ cd hadoop\-0.23.1
$ ./bin/hadoop distcp
12/02/09 17:40:23 ERROR tools.DistCp: Invalid arguments: 
java.lang.IllegalArgumentException: Target path not specified
	at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
	at org.apache.hadoop.tools.DistCp.run(DistCp.java:95)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
	at org.apache.hadoop.tools.DistCp.main(DistCp.java:357)
Invalid arguments: Target path not specified
usage: distcp OPTIONS [source_path...] <target_path>
              OPTIONS
 -async                 Should distcp execution be blocking
 -atomic                Commit all changes or none
 -bandwidth <arg>       Specify bandwidth per map in MB
 -delete                Delete from target, files missing in source
 -f <arg>               List of files that need to be copied
 -filelimit <arg>       (Deprecated!) Limit number of files copied to <= n
 -i                     Ignore failures during copy
 -log <arg>             Folder on DFS where distcp execution logs are
                        saved
 -m <arg>               Max number of concurrent maps to use for copy
 -mapredSslConf <arg>   Configuration for ssl config file, to use with
                        hftps://
 -overwrite             Choose to overwrite target files unconditionally,
                        even if they exist.
 -p <arg>               preserve status (rbugp)(replication, block-size,
                        user, group, permission)
 -sizelimit <arg>       (Deprecated!) Limit number of files copied to <= n
                        bytes
 -skipcrccheck          Whether to skip CRC checks between source and
                        target paths.
 -strategy <arg>        Copy strategy to use. Default is dividing work
                        based on file sizes
 -tmp <arg>             Intermediate work path to be used for atomic
                        commit
 -update                Update target, copying only missingfiles or
                        directories
$ ./bin/mapred distcp
12/02/09 17:40:29 ERROR tools.DistCp: Invalid arguments: 
java.lang.IllegalArgumentException: Target path not specified
	at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:86)
	at org.apache.hadoop.tools.DistCp.run(DistCp.java:95)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
	at org.apache.hadoop.tools.DistCp.main(DistCp.java:357)
Invalid arguments: Target path not specified
usage: distcp OPTIONS [source_path...] <target_path>
              OPTIONS
 -async                 Should distcp execution be blocking
 -atomic                Commit all changes or none
 -bandwidth <arg>       Specify bandwidth per map in MB
 -delete                Delete from target, files missing in source
 -f <arg>               List of files that need to be copied
 -filelimit <arg>       (Deprecated!) Limit number of files copied to <= n
 -i                     Ignore failures during copy
 -log <arg>             Folder on DFS where distcp execution logs are
                        saved
 -m <arg>               Max number of concurrent maps to use for copy
 -mapredSslConf <arg>   Configuration for ssl config file, to use with
                        hftps://
 -overwrite             Choose to overwrite target files unconditionally,
                        even if they exist.
 -p <arg>               preserve status (rbugp)(replication, block-size,
                        user, group, permission)
 -sizelimit <arg>       (Deprecated!) Limit number of files copied to <= n
                        bytes
 -skipcrccheck          Whether to skip CRC checks between source and
                        target paths.
 -strategy <arg>        Copy strategy to use. Default is dividing work
                        based on file sizes
 -tmp <arg>             Intermediate work path to be used for atomic
                        commit
 -update                Update target, copying only missingfiles or
                        directories
$ 
{noformat}

Normally the distcp jar is located via TOOL_PATH which is setup in hadoop-config.sh if not
overridden in the environment.  In both the {{hadoop}} and {{mapred}} cases I'm seeing TOOL_PATH
set via hadoop-config.sh.  It'd be interesting to see how one script is picking up the proper
TOOL_PATH but the other isn't (e.g.: prepend {{bash -x}} to each shell command and compare
the output)
                
> hadoop distcp fails to run with java.lang.NoClassDefFoundError
> --------------------------------------------------------------
>
>                 Key: MAPREDUCE-3845
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: distcp
>    Affects Versions: 0.23.1
>            Reporter: Roman Shaposhnik
>            Assignee: Roman Shaposhnik
>             Fix For: 0.23.1
>
>
> Here's how to reproduce:
> {noformat}
> $ cd /tmnp
> $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar
xzvf -
> $ cd hadoop-0.23.1
> $ ./bin/hadoop distcp
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class: org.apache.hadoop.tools.DistCp.  Program will exit.
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message