hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Forsberg <forsb...@opera.com>
Subject Merging of the local FS files threw an exception: java.io.IOException: java.lang.RuntimeException: java.io.EOFException
Date Wed, 20 Oct 2010 17:49:09 GMT
Hi!

I'm running Cloudera CDH2 update 2 (hadoop-0.20 0.20.1+169.113), and
after the upgrade I'm getting the following error in the reducers
during the copy phase in one of my larger jobs:

2010-10-20 17:43:22,343 INFO org.apache.hadoop.mapred.ReduceTask:
Initiating in-memory merge with 12 segments... 2010-10-20 17:43:22,344
INFO org.apache.hadoop.mapred.Merger: Merging 12 sorted segments
2010-10-20 17:43:22,344 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 12 segments left of total size: 382660295
bytes 2010-10-20 17:43:22,517 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_201010201640_0001_r_000000_0 Merging of the local FS files
threw an exception: java.io.IOException: java.lang.RuntimeException:
java.io.EOFException at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:128)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
at
org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:136)
at
org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350) at
org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156) at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2529)
Caused by: java.io.EOFException at
java.io.DataInputStream.readFully(DataInputStream.java:180) at
org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:154)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:125) ...
7 more

	at
	org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2533)

2010-10-20 17:43:22,765 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_201010201640_0001_r_000000_0 Merge of the inmemory files threw
an exception: java.io.IOException: Intermediate merge failed at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2651)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2576)
Caused by: java.lang.RuntimeException: java.io.EOFException at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:128)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
at
org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:139)
at
org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350) at
org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156) at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2635)
...
1 more Caused by: java.io.EOFException at
java.io.DataInputStream.readFully(DataInputStream.java:180) at
org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:154)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:122) ...
8 more

2010-10-20 17:43:23,766 INFO org.apache.hadoop.mapred.ReduceTask:
GetMapEventsThread exiting 2010-10-20 17:43:23,767 INFO
org.apache.hadoop.mapred.ReduceTask: getMapsEventsThread joined.
2010-10-20 17:43:23,774 INFO org.apache.hadoop.mapred.ReduceTask:
Closed ram manager 2010-10-20 17:43:23,931 WARN
org.apache.hadoop.mapred.TaskTracker: Error running child
java.io.IOException: Task: attempt_201010201640_0001_r_000000_0 - The
reduce copier failed at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380) at
org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by:
java.io.IOException: Intermediate merge failed at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2651)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2576)
Caused by: java.lang.RuntimeException: java.io.EOFException at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:128)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
at
org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:139)
at
org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350) at
org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156) at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2635)
...
1 more Caused by: java.io.EOFException at
java.io.DataInputStream.readFully(DataInputStream.java:180) at
org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:154)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:122) ...
8 more 2010-10-20 17:43:23,936 INFO
org.apache.hadoop.mapred.TaskRunner: Runnning cleanup for the task

I've tried:

*) Running with and without compressed map output, no difference.
*) With -Xmx512m and -Xmx768m, no difference.
*) Decreasing number of mappers and reducers on all nodes to decrease
   overall load.
*) Decreasing mapred.reduce.parallel.copies from 16 to 5 (default)

Any ideas on what to try next? What other information is required to
find out what's wrong?

Regards,
\EF
-- 
Erik Forsberg <forsberg@opera.com>
Developer, Opera Software - http://www.opera.com/

Mime
View raw message