hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng, Kai" <kai.zh...@intel.com>
Subject RE: Distcp fails with "Got EOF but currentPos = 240377856 < filelength = 1026034162" error
Date Tue, 19 Jan 2016 03:09:54 GMT
Looks like a file it’s copying is ended unexpectedly. Maybe need to find out which file,
check or read the file in other means to ensure it’s fine not being corrupt.


From: Buntu Dev [mailto:buntudev@gmail.com]
Sent: Tuesday, January 19, 2016 5:46 AM
To: user@hadoop.apache.org
Subject: Distcp fails with "Got EOF but currentPos = 240377856 < filelength = 1026034162"

I'm using distcp with these options to copy a hdfs directory from one cluster to another:

hadoop distcp -prb -i -update -skipcrccheck -delete hftp://cluster1/user/hive/warehouse/dir1/

I keep running into these errors related to EOF, what could be causing these errors and how
to fix this:

Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException:
Got EOF but currentPos = 240377856 < filelength = 1026034162
            at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:289)
            at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:257)
            at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:184)
            at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:124)
            at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100)
            at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
            ... 11 more

Also I'm using the '-i' to ignore and continue on failures but the distcp does retry 3 times
and stops. Can anyone throw some light on what else could be going wrong.

View raw message