hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From java8964 java8964 <java8...@hotmail.com>
Subject RE: distcp failed "Copy failed: ENOENT: No such file or directory"
Date Fri, 06 Sep 2013 20:38:02 GMT
The error doesn't mean the file not existed in the HDFS, but it means local disk. If you read
the error stack trace:
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:581)
It indicates the error happened on Local file system.
If you try to copy data from an existing cluster to a new setup cluster, then It most likely
some misconfiguration in your new cluster causing this.

When you use the distcp command, hadoop will generate a lot of MR jobs to copy the data in
Mapper tasks. (No reducers in this case). You should check the JT to find out which task node
gave this error, and check the log to identify the root cause, especially check the local.dir
or log.dir settings in the new cluster.

Date: Fri, 6 Sep 2013 15:36:48 +0800
Subject: distcp failed "Copy failed: ENOENT: No such file or directory"
From: justlooks@gmail.com
To: user@hadoop.apache.org

# sudo -u hdfs hadoop distcp hdfs:// hdfs://
13/09/06 15:34:11 INFO tools.DistCp: srcPaths=[hdfs://]

13/09/06 15:34:11 INFO tools.DistCp: destPath=hdfs://
13/09/06 15:34:12 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id

13/09/06 15:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker,
13/09/06 15:34:12 WARN conf.Configuration: slave.host.name is deprecated. Instead, use dfs.datanode.hostname

With failures, global counters are inaccurate; consider running with -i
Copy failed: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:581)

        at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:427)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:579)
        at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1038)

        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

        at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
the src and dest dir is all exist
# sudo -u hdfs hadoop fs -ls /alex
Found 3 items
drwxr-xr-x   - root supergroup          0 2013-08-16 10:14 /alex/mydump
drwxr-xr-x   - root supergroup          0 2013-08-28 10:48 /alex/tdump
drwxr-xr-x   - root supergroup          0 2013-08-28 10:54 /alex/tdump2

#  sudo -u hdfs hadoop fs -ls /benchmarks
Found 1 items
drwxr-xr-x   - hdfs hdfs          0 2013-08-23 17:02 /benchmarks/TestDFSIO

View raw message