hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ayon Sinha <ayonsi...@yahoo.com>
Subject distcp -update option problem
Date Tue, 23 Aug 2011 22:48:49 GMT
When I use -overwrite everything gets copied over fine. And the files are not corrupt.

When I use the -update option for distcp, I constantly get this WARN + exception. What is
it trying to do and what is failing?

11/08/23 22:43:06 WARN hdfs.DFSClient: src=/analytics_hive_tables/web_etl_tables/mass_actions_table/mass_actions_table_2011-01-01-2011-08-16.out,
datanodes[0].getName()=10.177.1.218:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:298)
at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:782)
at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:719)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:553)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:53)
at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1261)
at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1120)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)

 
-Ayon
See My Photos on Flickr
Also check out my Blog for answers to commonly asked questions.

Mime
View raw message