Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 45824 invoked from network); 30 Oct 2007 23:42:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 30 Oct 2007 23:42:12 -0000 Received: (qmail 89953 invoked by uid 500); 30 Oct 2007 23:41:59 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 89913 invoked by uid 500); 30 Oct 2007 23:41:59 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 89904 invoked by uid 99); 30 Oct 2007 23:41:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Oct 2007 16:41:59 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Oct 2007 23:42:11 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id C0C3771420E for ; Tue, 30 Oct 2007 16:41:50 -0700 (PDT) Message-ID: <29797009.1193787710787.JavaMail.jira@brutus> Date: Tue, 30 Oct 2007 16:41:50 -0700 (PDT) From: "Murtaza A. Basrai (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Created: (HADOOP-2129) distcp between two clusters does not work if it is run on the target cluster MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org distcp between two clusters does not work if it is run on the target cluster ---------------------------------------------------------------------------- Key: HADOOP-2129 URL: https://issues.apache.org/jira/browse/HADOOP-2129 Project: Hadoop Issue Type: Bug Components: util Affects Versions: 0.16.0 Environment: Nightly build: http://hadoopqa.yst.corp.yahoo.com:8080/hudson/job/Hadoop-LinuxTest/718/ With patches for HADOOP-2033 and HADOOP-2048. Reporter: Murtaza A. Basrai Priority: Critical I am trying to copy a directory (~100k files, ~500GB) between two clusters A and B (~70 nodes), using a command like: hadoop distcp -log /logdir hdfs://namenode-of-A:8600/srcdir hdfs://namenode-of-B:8600/targetdir I tried 4 ways of doing it: 1) Copy from A to B, by running distcp on A 2) Copy from A to B, by running distcp on B 3) Copy from B to A, by running distcp on B 4) Copy from B to A, by running distcp on A Invocations 1 and 3 succeeded, but 2 and 4 failed. I got a lot of errors of the type below: 07/10/30 20:52:11 INFO mapred.JobClient: Running job: job_200710180049_0115 07/10/30 20:52:12 INFO mapred.JobClient: map 0% reduce 0% 07/10/30 20:54:41 INFO mapred.JobClient: map 1% reduce 0% 07/10/30 20:56:52 INFO mapred.JobClient: map 2% reduce 0% 07/10/30 20:57:41 INFO mapred.JobClient: Task Id : task_200710180049_0115_m_000184_0, Status : FAILED java.io.IOException: Some copies could not complete. See log for details. at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:407) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760) followed by the job failing: 07/10/30 22:07:41 INFO mapred.JobClient: map 99% reduce 100% Copy failed: java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:688) at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:481) at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:555) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:67) at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:566) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.