hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varene Olivier <var...@echo.fr>
Subject Mapper Reducer : Unit Test and mocking with static variables
Date Thu, 27 May 2010 08:40:36 GMT
Hello to all,

first of all many thanks for this great piece of software you are all
contributing to :)

I am actually creating a program in Hadoop MapReduce, but I intend to
massively use JUnit Tests to validate my code.

To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
function with it, but inside my Mapper.map function, I am using Counters 
to maintain internal statistics. The update of those counters make the 
test fails with a null pointer (JavaNullPointerException).
There is an issue "Counters" of variable visibility from my test ...

Do you have any idea how to mock/validate/overcome this issue ?

Thanks a lots
Best Regards
Olivier Varene

: Code :

private static enum Counters {NBLINES};
private static IntWritable one = new IntWritable(1);
private static LongWritable oneL = new LongWritable(1);

// Mapper
  * outputs 1 for each line encountered
public static class LCMapper extends
	Mapper<LongWritable, Text, IntWritable, LongWritable>
   private final static IntWritable one = new IntWritable(1);
   private final static LongWritable oneL = new LongWritable(1);
   * for each line encountered outputs 1 for the key 1
   public void map(LongWritable p_key, Text p_inputs, Context p_context)
	throws IOException, InterruptedException
   } // end map
} // end Mapper

: Test :

public void testLCMapper() throws IOException, InterruptedException
   LCMapper mapper = new LCMapper();
   Text value = new Text("ligneA\nlignB\n");
   // mock the map job execution context
   Context context = mock(Context.class);
   try {
     mapper.map(null, value, context);
   } catch (NullPointerException e)
     // exception raised by context.getCounter(...) null pointer
     // HOW TO MOCK ???
     // Counter is a static enum from the englobing class ...
     // issue with variable visibility (from map function) in this test
   // verify that the Mapper issued (one,oneL) pair
} // end testLCMapper

View raw message