hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Buntu Dev <buntu...@gmail.com>
Subject Distcp fails with "Got EOF but currentPos = 240377856 < filelength = 1026034162" error
Date Mon, 18 Jan 2016 21:45:53 GMT
I'm using distcp with these options to copy a hdfs directory from one
cluster to another:

~~~~
hadoop distcp -prb -i -update -skipcrccheck -delete
hftp://cluster1/user/hive/warehouse/dir1/ hdfs://cluster2/dir1/
~~~~

I keep running into these errors related to EOF, what could be causing
these errors and how to fix this:

~~~~~~~~~
Caused by:
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException:
java.io.IOException: Got EOF but currentPos = 240377856 < filelength =
1026034162
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:289)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:257)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:184)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:124)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
~~~~~~~~~~


Also I'm using the '-i' to ignore and continue on failures but the distcp
does retry 3 times and stops. Can anyone throw some light on what else
could be going wrong.


Thanks!

Mime
View raw message