hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ed <hadoopn...@gmail.com>
Subject How to stop a mapper within a map-reduce job when you detect bad input
Date Tue, 19 Oct 2010 22:44:52 GMT

I have a simple map-reduce job that reads in zipped files and converts them
to lzo compression.  Some of the files are not properly zipped which results
in Hadoop throwing an "java.io.EOFException: Unexpected end of input stream
error" and causes the job to fail.  Is there a way to catch this exception
and tell hadoop to just ignore the file and move on?  I think the exception
is being thrown by the class reading in the Gzip file and not my mapper
class.  Is this correct?  Is there a way to handle this type of error

Thank you!


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message