hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varene Olivier <var...@echo.fr>
Subject Re: Mapper Reducer : Unit Test and mocking with static variables
Date Fri, 28 May 2010 06:38:14 GMT
Thanks a lot Aaron,

I was looking into it, I just hope, it will be compliant with 0.20.2 ...

Otherwise, I am stuck :(

Cheers
Olivier

Aaron Kimball a écrit :
> Varene,
> 
> You might want to check out MRUnit. It's a unit test harness that contains
> mock objects for the context & other associated classes, and works with
> JUnit.
> 
> It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
> Distribution for Hadoop. See
> http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation.
> 
> Cheers,
> - Aaron
> 
> On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <varene@echo.fr> wrote:
> 
>> FYI
>> I am on Hadoop 0.20.2, JUnit 4
>>
>>
>> Varene Olivier a écrit :
>>
>>  Hello to all,
>>> first of all many thanks for this great piece of software you are all
>>> contributing to :)
>>>
>>> I am actually creating a program in Hadoop MapReduce, but I intend to
>>> massively use JUnit Tests to validate my code.
>>>
>>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>>> function with it, but inside my Mapper.map function, I am using Counters
>>> to maintain internal statistics. The update of those counters make the test
>>> fails with a null pointer (JavaNullPointerException).
>>> There is an issue "Counters" of variable visibility from my test ...
>>>
>>> Do you have any idea how to mock/validate/overcome this issue ?
>>>
>>>
>>>
>>> Thanks a lots
>>> Best Regards
>>> Olivier Varene
>>>
>>>
>>> : Code :
>>> --------
>>>
>>> private static enum Counters {NBLINES};
>>> private static IntWritable one = new IntWritable(1);
>>> private static LongWritable oneL = new LongWritable(1);
>>>
>>>
>>> // Mapper
>>> /**
>>>  * outputs 1 for each line encountered
>>>  */
>>> public static class LCMapper extends
>>>    Mapper<LongWritable, Text, IntWritable, LongWritable>
>>> {
>>>  private final static IntWritable one = new IntWritable(1);
>>>  private final static LongWritable oneL = new LongWritable(1);
>>>  /**
>>>  * for each line encountered outputs 1 for the key 1
>>>  */
>>>  public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>>    throws IOException, InterruptedException
>>>  {
>>>    p_context.getCounter(Counters.NBLINES).increment(1);
>>>    p_context.write(one,oneL);     } // end map
>>> } // end Mapper
>>>
>>> : Test :
>>> --------
>>>
>>> @Test
>>> public void testLCMapper() throws IOException, InterruptedException
>>> {     LCMapper mapper = new LCMapper();
>>>  Text value = new Text("ligneA\nlignB\n");
>>>  // mock the map job execution context
>>>  Context context = mock(Context.class);
>>>  try {
>>>    mapper.map(null, value, context);
>>>  } catch (NullPointerException e)
>>>  {
>>>    // exception raised by context.getCounter(...) null pointer
>>>    // HOW TO MOCK ???
>>>    // Counter is a static enum from the englobing class ...
>>>    // issue with variable visibility (from map function) in this test
>>>  }
>>>  // verify that the Mapper issued (one,oneL) pair
>>>  verify(context).write(one,oneL);
>>>       } // end testLCMapper
>>>
>>>
>>>
> 

Mime
View raw message