hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Murtaza A. Basrai (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2129) distcp between two clusters does not work if it is run on the target cluster
Date Tue, 30 Oct 2007 23:41:50 GMT
distcp between two clusters does not work if it is run on the target cluster
----------------------------------------------------------------------------

                 Key: HADOOP-2129
                 URL: https://issues.apache.org/jira/browse/HADOOP-2129
             Project: Hadoop
          Issue Type: Bug
          Components: util
    Affects Versions: 0.16.0
         Environment: Nightly build: http://hadoopqa.yst.corp.yahoo.com:8080/hudson/job/Hadoop-LinuxTest/718/
With patches for HADOOP-2033 and HADOOP-2048.

            Reporter: Murtaza A. Basrai
            Priority: Critical


I am trying to copy a directory (~100k files, ~500GB) between two clusters A and B (~70 nodes),
using a command like:

hadoop distcp -log /logdir hdfs://namenode-of-A:8600/srcdir hdfs://namenode-of-B:8600/targetdir


I tried 4 ways of doing it:

1) Copy from A to B, by running distcp on A
2) Copy from A to B, by running distcp on B
3) Copy from B to A, by running distcp on B
4) Copy from B to A, by running distcp on A

Invocations 1 and 3 succeeded, but 2 and 4 failed.

I got a lot of errors of the type below:

07/10/30 20:52:11 INFO mapred.JobClient: Running job: job_200710180049_0115
07/10/30 20:52:12 INFO mapred.JobClient:  map 0% reduce 0%
07/10/30 20:54:41 INFO mapred.JobClient:  map 1% reduce 0%
07/10/30 20:56:52 INFO mapred.JobClient:  map 2% reduce 0%
07/10/30 20:57:41 INFO mapred.JobClient: Task Id : task_200710180049_0115_m_000184_0, Status
: FAILED
java.io.IOException: Some copies could not complete. See log for details.
        at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:407)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)

followed by the job failing:

07/10/30 22:07:41 INFO mapred.JobClient:  map 99% reduce 100%
Copy failed: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:688)
        at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:481)
        at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:555)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:54)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:67)
        at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:566)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message