hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hadoopman <hadoop...@gmail.com>
Subject HDFS and distcp issue??
Date Sat, 04 Dec 2010 20:47:52 GMT
I've run into an interesting problem with syncing a couple of clusters 
using distcp.  We've validated that it works to a local installation 
from our remote cluster.  I suspect our firewalls 'may' be responsible 
for the problem we're experiencing.  We're using ports 9000, 9001 and 
50010.I've verified all three ports are available to the namenodes and 
datanodes in both directions.  Is there something else we're missing?

Looks like it get's to 80% before it fails.  Here's what we're seeing.

# user@hnn1:~$ hadoop distcp hdfs://hnn1:9000/user/testing 
hdfs://hnn2:9000/user

10/12/03 15:58:10 INFO tools.DistCp: 
srcPaths=[hdfs://hnn1:9000/user/testing]

10/12/03 15:58:10 INFO tools.DistCp: destPath=hdfs://hnn2:9000/user

10/12/03 15:58:11 INFO tools.DistCp: srcCount=6

10/12/03 15:58:11 INFO mapred.JobClient: Running job: job_201011221457_0019

10/12/03 15:58:12 INFO mapred.JobClient:  map 0% reduce 0%

  10/12/03 15:58:36 INFO mapred.JobClient:  map 19% reduce 0%

10/12/03 15:58:45 INFO mapred.JobClient:  map 39% reduce 0%

10/12/03 15:59:03 INFO mapred.JobClient:  map 60% reduce 0%

10/12/03 15:59:12 INFO mapred.JobClient:  map 80% reduce 0%

10/12/03 15:59:32 INFO mapred.JobClient: Task Id : 
attempt_201011221457_0019_m_000000_0, Status : FAILED

java.io.IOException: Copied: 0 Skipped: 0 Failed: 5

         at 
org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:572)

         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)

         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)

         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)

         at org.apache.hadoop.mapred.Child.main(Child.java:170)

10/12/03 15:59:33 INFO mapred.JobClient:  map 0% reduce 0%

10/12/03 15:59:55 INFO mapred.JobClient:  map 19% reduce 0%

10/12/03 16:00:04 INFO mapred.JobClient:  map 39% reduce 0%

10/12/03 16:00:22 INFO mapred.JobClient:  map 60% reduce 0%

10/12/03 16:00:31 INFO mapred.JobClient:  map 80% reduce 0%

10/12/03 16:00:51 INFO mapred.JobClient: Task Id : 
attempt_201011221457_0019_m_000000_1, Status : FAILED

java.io.IOException: Copied: 0 Skipped: 0 Failed: 5

         at 
org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:572)

         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)

         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)

         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)

         at org.apache.hadoop.mapred.Child.main(Child.java:170)

Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message