hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wilkes, Chris" <cwil...@gmail.com>
Subject Should mapreduce.ReduceContext reuse same object in nextKeyValue?
Date Tue, 12 Jan 2010 23:53:25 GMT
I created my own Writable class to store 3 pieces of information.  In  
my mapreducer.Reducer class I collect all of them and then process as  
a group, ie:

reduce(key, values, context) {
   List<Foo> myFoos =new ArrayList();
   for (Foo value : values) {

I was perplexed when entries in the list changed underneath me so I  
put a freeze() method on it:

boolean m_frozen = false;
public void freeze() {
   m_frozen = true;
public void readFields(DataInput in) throws IOException {	
   if (m_frozen) {
      throw new IllegalStateException();

And noted that the exception was thrown:
         at Foo.readFields(Foo.java:169)
         at org.apache.hadoop.io.serializer.WritableSerialization 
         at org.apache.hadoop.io.serializer.WritableSerialization 
         at org.apache.hadoop.mapreduce.ReduceContext 

Am I doing something wrong?  Should I expect this VALUEIN object to  
change from underneath me?  I'm using hadoop 0.20.1 (from a cloudera  

View raw message