hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan LeCompte" <lecom...@gmail.com>
Subject Issue in reduce phase with SortedMapWritable and custom Writables as values
Date Tue, 09 Sep 2008 04:36:45 GMT

I'm attempting to use a SortedMapWritable with a LongWritable as the
key and a custom implementation of org.apache.hadoop.io.Writable as
the value. I notice that my program works fine when I use another
primitive wrapper (e.g. Text) as the value, but fails with the
following exception when I use my custom Writable instance:

2008-09-08 23:25:02,072 INFO org.apache.hadoop.mapred.ReduceTask:
Initiating in-memory merge with 1 segments...
2008-09-08 23:25:02,077 INFO org.apache.hadoop.mapred.Merger: Merging
1 sorted segments
2008-09-08 23:25:02,077 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 1 segments left of total size: 5492 bytes
2008-09-08 23:25:02,099 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200809082247_0005_r_000000_0 Merge of the inmemory files threw
n exception: java.io.IOException: Intermedate merge failed
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2133)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2064)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80)
        at org.apache.hadoop.io.SortedMapWritable.readFields(SortedMapWritable.java:179)

I noticed that the AbstractMapWritable class has a protected
"addToMap(Class clazz)" method. Do I somehow need to let my
SortedMapWritable instance know about my custom Writable value? I've
properly implemented the custom Writable object (it just contains a
few primitives, like longs and ints).

Any insight is appreciated.


View raw message