hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wilkes, Chris" <cwil...@gmail.com>
Subject Should mapreduce.ReduceContext reuse same object in nextKeyValue?
Date Tue, 12 Jan 2010 23:53:25 GMT
I created my own Writable class to store 3 pieces of information.  In  
my mapreducer.Reducer class I collect all of them and then process as  
a group, ie:

reduce(key, values, context) {
   List<Foo> myFoos =new ArrayList();
   for (Foo value : values) {
    myFoos.add(value);
   }
}

I was perplexed when entries in the list changed underneath me so I  
put a freeze() method on it:

boolean m_frozen = false;
public void freeze() {
   m_frozen = true;
}
@Override
public void readFields(DataInput in) throws IOException {	
   if (m_frozen) {
      throw new IllegalStateException();
   }
}

And noted that the exception was thrown:
         at Foo.readFields(Foo.java:169)
         at org.apache.hadoop.io.serializer.WritableSerialization 
$WritableDeserializer.deserialize(WritableSerialization.java:67)
         at org.apache.hadoop.io.serializer.WritableSerialization 
$WritableDeserializer.deserialize(WritableSerialization.java:40)
         at  
org 
.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java: 
116)
         at org.apache.hadoop.mapreduce.ReduceContext 
$ValueIterator.next(ReduceContext.java:163)

Am I doing something wrong?  Should I expect this VALUEIN object to  
change from underneath me?  I'm using hadoop 0.20.1 (from a cloudera  
tarball)

Chris
Mime
View raw message