Return-Path: Delivered-To: apmail-hadoop-core-dev-archive@www.apache.org Received: (qmail 26471 invoked from network); 19 Jun 2009 11:55:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 19 Jun 2009 11:55:23 -0000 Received: (qmail 56774 invoked by uid 500); 19 Jun 2009 11:55:33 -0000 Delivered-To: apmail-hadoop-core-dev-archive@hadoop.apache.org Received: (qmail 56719 invoked by uid 500); 19 Jun 2009 11:55:32 -0000 Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-dev@hadoop.apache.org Received: (qmail 56682 invoked by uid 99); 19 Jun 2009 11:55:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 11:55:32 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 11:55:29 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 68896234C1EE for ; Fri, 19 Jun 2009 04:55:08 -0700 (PDT) Message-ID: <2029084550.1245412508427.JavaMail.jira@brutus> Date: Fri, 19 Jun 2009 04:55:08 -0700 (PDT) From: "Ravi Gummadi (JIRA)" To: core-dev@hadoop.apache.org Subject: [jira] Assigned: (HADOOP-6072) distcp should place the file distcp_src_files in distributed cache In-Reply-To: <1587886309.1245268447882.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi reassigned HADOOP-6072: ------------------------------------ Assignee: Ravi Gummadi > distcp should place the file distcp_src_files in distributed cache > ------------------------------------------------------------------ > > Key: HADOOP-6072 > URL: https://issues.apache.org/jira/browse/HADOOP-6072 > Project: Hadoop Core > Issue Type: Improvement > Components: tools/distcp > Affects Versions: 0.21.0 > Reporter: Ravi Gummadi > Assignee: Ravi Gummadi > Fix For: 0.21.0 > > > When large number of files are being copied by distcp, accessing distcp_src_files seems to be an issue, as all map tasks would be accessing this file. The error message seen is: > 09/06/16 10:13:16 INFO mapred.JobClient: Task Id : attempt_200906040559_0110_m_003348_0, Status : FAILED > java.io.IOException: Could not obtain block: blk_-4229860619941366534_1500174 > file=/mapredsystem/hadoop/mapredsystem/distcp_7fiyvq/_distcp_src_files > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1757) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1585) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1712) > at java.io.DataInputStream.readFully(DataInputStream.java:178) > at java.io.DataInputStream.readFully(DataInputStream.java:152) > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1428) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1417) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1412) > at org.apache.hadoop.mapred.SequenceFileRecordReader.(SequenceFileRecordReader.java:43) > at org.apache.hadoop.tools.DistCp$CopyInputFormat.getRecordReader(DistCp.java:299) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:336) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > This could be because of HADOOP-6038 and/or HADOOP-4681. > If distcp places this special file distcp_src_files in distributed cache, that could solve the problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.