hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devaraj Das <d...@yahoo-inc.com>
Subject Re: "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Date Wed, 29 Oct 2008 04:00:44 GMT
Quick question (I haven't looked at your comparator code yet) - is this
reproducible/consistent?


On 10/28/08 11:52 PM, "Deepika Khera" <deepikak@collarity.com> wrote:

> I am getting a similar exception too with Hadoop 0.18.1(See stacktrace
> below), though its an EOFException. Does anyone have any idea about what
> it means and how it can be fixed?
> 
> 2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810241922_0844_r_000006_0 Merge of the inmemory files threw
> an exception: java.io.IOException: Intermedate merge failed
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
> MemMerge(ReduceTask.java:2147)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
> ReduceTask.java:2078)
> Caused by: java.lang.RuntimeException: java.io.EOFException
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 103)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
> at
> org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:122)
> at
> org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:49)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:321)
> at org.apache.hadoop.mapred.Merger.merge(Merger.java:72)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
> MemMerge(ReduceTask.java:2123)
> ... 1 more
> Caused by: java.io.EOFException
> at
> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
> at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
> at
> com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
> ey.java:40)
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 97)
> ... 7 more
> 
> 2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810241922_0844_r_000006_0 Merging of the local FS files threw
> an exception: java.io.IOException: java.lang.RuntimeException:
> java.io.EOFException
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 103)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
> at
> org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:135)
> at
> org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:102)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.ja
> va:226)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
> at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
> eTask.java:2035)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
> at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
> at
> com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
> ey.java:40)
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 97)
> ... 7 more
> 
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
> eTask.java:2039)
> 
> 2008-10-27 16:53:07,907 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> java.io.IOException: attempt_200810241922_0844_r_000006_0The reduce
> copier failed
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> 
> 
> 
> 
> My WritableComparable class looks like this -
> 
> 
> 
> 
> public class IncrementalURLIndexKey implements WritableComparable {
>   private UTF8 url;
>   private long userid;
> 
>   public IncrementalURLIndexKey() {
>   }
> 
>   public IncrementalURLIndexKey(UTF8 url, long userid) {
>     this.url = url;
>     this.userid = userid;
>   }
> 
>   public UTF8 getUrl() {
>     return url;
>   }
> 
>   public long getUserid() {
>     return userid;
>   }
> 
>   public void write(DataOutput out) throws IOException {
>     IOUtil.writeUTF8(out, url);
>     out.writeLong(userid);
>   }
> 
>   public void readFields(DataInput in) throws IOException {
>     url = IOUtil.readUTF8(in, url);
>     userid = in.readLong();
>   }
> 
>   public int compareTo(Object o) {
>     IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
>     int result = url.compareTo(other.getUrl());
>     if (result == 0) result = CUID.compare(userid, other.userid);
>     return result;
>   }
> 
>   /**
>    * A Comparator optimized for IncrementalURLIndexKey.
>    */
>   public static class GroupingComparator extends WritableComparator {
>     public GroupingComparator() {
>       super(IncrementalURLIndexKey.class, true);
>     }
> 
>     public int compare(WritableComparable a, WritableComparable b) {
>       IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
>       IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;
> 
>       return key1.getUrl().compareTo(key2.getUrl());
>     }
>   }
> 
>   static {
>     WritableComparator.define(IncrementalURLIndexKey.class, new
> GroupingComparator());
>   }
> }
> 
> 
> Thanks,
> Deepika
> 
> -----Original Message-----
> From: Grant Ingersoll [mailto:gsingers@apache.org]
> Sent: Monday, October 27, 2008 7:14 PM
> To: core-user@hadoop.apache.org
> Subject: Re: "Merge of the inmemory files threw an exception" and diffs
> between 0.17.2 and 0.18.1
> 
> I suppose I should add an actual question:  Is this a bug or are we
> relying on a "feature" that has since been fixed?
> 
> On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:
> 
>> Hi,
>> 
>> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
>> some of our clustering code and Hadoop 0.18.1.  The thread in
>> context is at:  http://mahout.markmail.org/message/vcyvlz2met7fnthr
>> 
>> The problem seems to occur when going from 0.17.2 to 0.18.1.  In the
>> user logs, we are seeing the following exception:
>> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
>> to the last merge-pass, with 2 segments left of total size: 5011 bytes
>> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
>> threw an exception: java.io.IOException: Intermedate merge failed
>>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
>>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.run(ReduceTask.java:2078)
>> Caused by: java.lang.NumberFormatException: For input string: "["
>>        at  
>> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
>> 1224)
>>        at java.lang.Double.parseDouble(Double.java:510)
>>        at  
>> org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
>>        at  
>> org 
>> .apache 
>> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
>>        at  
>> org 
>> .apache 
>> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
>> 38)
>>        at  
>> org 
>> .apache 
>> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
>> 31)
>>        at org.apache.hadoop.mapred.ReduceTask
>> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
>>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
>> $3100(ReduceTask.java:341)
>>        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>> 
>> And in the main output log (from running  bin/hadoop jar  mahout/
>> examples/build/apache-mahout-examples-0.1-dev.job
>> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
>> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
>> attempt_200810272112_0011_r_000000_0, Status : FAILED
>> java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
>> copier failed
>>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
>>        at org.apache.hadoop.mapred.TaskTracker
>> $Child.main(TaskTracker.java:2207)
>> 
>> If I run this exact same job on 0.17.2 it all runs fine.  I suppose
>> either a bug was introduced in 0.18.1 or a bug was fixed that we
>> were relying on.  Looking at the release notes between the fixes,
>> nothing in particular struck me as related.  If it helps, I can
>> provide the instructions for how to run the example in question
>> (they need to be written up anyway!)
>> 
>> 
>> I see some related things at
> http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
> n+exception 
>> , but those are older, it seems, so not sure what to make of them.
>> 
>> Thanks,
>> Grant
> 
> 



Mime
View raw message