hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varene Olivier <var...@echo.fr>
Subject Re: Mapper Reducer : Unit Test and mocking with static variables
Date Thu, 27 May 2010 16:13:03 GMT
FYI
I am on Hadoop 0.20.2, JUnit 4


Varene Olivier a écrit :
> Hello to all,
> 
> first of all many thanks for this great piece of software you are all
> contributing to :)
> 
> I am actually creating a program in Hadoop MapReduce, but I intend to
> massively use JUnit Tests to validate my code.
> 
> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
> function with it, but inside my Mapper.map function, I am using Counters 
> to maintain internal statistics. The update of those counters make the 
> test fails with a null pointer (JavaNullPointerException).
> There is an issue "Counters" of variable visibility from my test ...
> 
> Do you have any idea how to mock/validate/overcome this issue ?
> 
> 
> 
> Thanks a lots
> Best Regards
> Olivier Varene
> 
> 
> : Code :
> --------
> 
> private static enum Counters {NBLINES};
> private static IntWritable one = new IntWritable(1);
> private static LongWritable oneL = new LongWritable(1);
> 
> 
> // Mapper
> /**
>  * outputs 1 for each line encountered
>  */
> public static class LCMapper extends
>     Mapper<LongWritable, Text, IntWritable, LongWritable>
> {
>   private final static IntWritable one = new IntWritable(1);
>   private final static LongWritable oneL = new LongWritable(1);
>   /**
>   * for each line encountered outputs 1 for the key 1
>   */
>   public void map(LongWritable p_key, Text p_inputs, Context p_context)
>     throws IOException, InterruptedException
>   {
>     p_context.getCounter(Counters.NBLINES).increment(1);
>     p_context.write(one,oneL);   
>   } // end map
> } // end Mapper
> 
> : Test :
> --------
> 
> @Test
> public void testLCMapper() throws IOException, InterruptedException
> {   
>   LCMapper mapper = new LCMapper();
>   Text value = new Text("ligneA\nlignB\n");
>   // mock the map job execution context
>   Context context = mock(Context.class);
>   try {
>     mapper.map(null, value, context);
>   } catch (NullPointerException e)
>   {
>     // exception raised by context.getCounter(...) null pointer
>     // HOW TO MOCK ???
>     // Counter is a static enum from the englobing class ...
>     // issue with variable visibility (from map function) in this test
>   }
>   // verify that the Mapper issued (one,oneL) pair
>   verify(context).write(one,oneL);
>        
> } // end testLCMapper
> 
> 

Mime
View raw message