hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball" <aa...@cloudera.com>
Subject Re: EOFException causes failure.
Date Mon, 05 Jan 2009 04:22:24 GMT
It seems to be having a problem deserializing data from a SequenceFile. It's
the same task failing over and over again -- the same slice of a particular
SequenceFile generated by the previous iteration. This suggests that maybe
you are generating incorrect SequenceFiles in the previous iteration.

Are you using custom implementations of Writable? Maybe your readFields()
and write() methods do not line up with one another correctly? Or that
there's a bug where you claim to have a variable-length entry of K bytes,
but are actually writing fewer than K?

Good luck,
- Aaron

On Sat, Jan 3, 2009 at 9:39 PM, gziyang <gziyang@gmail.com> wrote:

> Hello,
>
> I would like to run a mapreduce program on hadoop 0.19.0. The program is
> designed to iterate for 1000 times. However, in a certain iteration, the
> program fails due to an EOFException, but the jobs complete for all the
> previous iterations. Sometimes the program fails in the 5th iteration, while
> in other times, the program might fail until the 9th iteration. It seems a
> little random.
>
> A similar situation has been discussed for hadoop version 0.2, but it is
> said the bug has been fixed in version 0.3. Anybody knows that what is wrong
> with my hadoop 0.19.0?
>
> ...
> 09/01/04 12:54:37 INFO mapred.JobClient:  map 60% reduce 0%
> 09/01/04 12:54:41 INFO mapred.JobClient:  map 66% reduce 0%
> 09/01/04 12:54:41 INFO mapred.JobClient: Task Id :
> attempt_200901041244_0007_m_000011_0, Status : FAILED
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>        at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
>        at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
>        at org.apache.hadoop.mapred.Child.main(Child.java:155)
> 09/01/04 12:54:43 INFO mapred.JobClient:  map 70% reduce 1%
> 09/01/04 12:54:45 INFO mapred.JobClient:  map 78% reduce 1%
> 09/01/04 12:54:46 INFO mapred.JobClient:  map 81% reduce 2%
> 09/01/04 12:54:48 INFO mapred.JobClient:  map 83% reduce 2%
> 09/01/04 12:54:48 INFO mapred.JobClient: Task Id :
> attempt_200901041244_0007_m_000011_1, Status : FAILED
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>        at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
>        at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
>        at org.apache.hadoop.mapred.Child.main(Child.java:155)
> 09/01/04 12:54:50 INFO mapred.JobClient:  map 83% reduce 3%
> 09/01/04 12:54:51 INFO mapred.JobClient:  map 86% reduce 4%
> 09/01/04 12:54:53 INFO mapred.JobClient:  map 86% reduce 6%
> 09/01/04 12:54:53 INFO mapred.JobClient: Task Id :
> attempt_200901041244_0007_m_000011_2, Status : FAILED
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readFully(DataInputStream.java:152)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
>        at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
>        at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:58)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:321)
>        at org.apache.hadoop.mapred.Child.main(Child.java:155)
> 09/01/04 12:54:55 INFO mapred.JobClient:  map 86% reduce 9%
> 09/01/04 12:54:56 INFO mapred.JobClient:  map 88% reduce 10%
> 09/01/04 12:54:58 INFO mapred.JobClient:  map 88% reduce 15%
> java.io.IOException: Job failed!
>
> Thanks a lot!
>
> 2009-01-04
>
>
>
> gziyang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message