hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varene Olivier <var...@echo.fr>
Subject Re: Mapper Reducer : Unit Test and mocking with static variables
Date Fri, 28 May 2010 12:55:02 GMT
FYI

MRUnit is the way to go and it is compatible (as far as I am aware of) 
with the 0.20.2

Cheers
Olivier

Varene Olivier a écrit :
> Thanks a lot Aaron,
> 
> I was looking into it, I just hope, it will be compliant with 0.20.2 ...
> 
> Otherwise, I am stuck :(
> 
> Cheers
> Olivier
> 
> Aaron Kimball a écrit :
>> Varene,
>>
>> You might want to check out MRUnit. It's a unit test harness that 
>> contains
>> mock objects for the context & other associated classes, and works with
>> JUnit.
>>
>> It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
>> Distribution for Hadoop. See
>> http://archive.cloudera.com/docs/mrunit/index.html for MRUnit 
>> documentation.
>>
>> Cheers,
>> - Aaron
>>
>> On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <varene@echo.fr> wrote:
>>
>>> FYI
>>> I am on Hadoop 0.20.2, JUnit 4
>>>
>>>
>>> Varene Olivier a écrit :
>>>
>>>  Hello to all,
>>>> first of all many thanks for this great piece of software you are all
>>>> contributing to :)
>>>>
>>>> I am actually creating a program in Hadoop MapReduce, but I intend to
>>>> massively use JUnit Tests to validate my code.
>>>>
>>>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>>>> function with it, but inside my Mapper.map function, I am using 
>>>> Counters
>>>> to maintain internal statistics. The update of those counters make 
>>>> the test
>>>> fails with a null pointer (JavaNullPointerException).
>>>> There is an issue "Counters" of variable visibility from my test ...
>>>>
>>>> Do you have any idea how to mock/validate/overcome this issue ?
>>>>
>>>>
>>>>
>>>> Thanks a lots
>>>> Best Regards
>>>> Olivier Varene
>>>>
>>>>
>>>> : Code :
>>>> --------
>>>>
>>>> private static enum Counters {NBLINES};
>>>> private static IntWritable one = new IntWritable(1);
>>>> private static LongWritable oneL = new LongWritable(1);
>>>>
>>>>
>>>> // Mapper
>>>> /**
>>>>  * outputs 1 for each line encountered
>>>>  */
>>>> public static class LCMapper extends
>>>>    Mapper<LongWritable, Text, IntWritable, LongWritable>
>>>> {
>>>>  private final static IntWritable one = new IntWritable(1);
>>>>  private final static LongWritable oneL = new LongWritable(1);
>>>>  /**
>>>>  * for each line encountered outputs 1 for the key 1
>>>>  */
>>>>  public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>>>    throws IOException, InterruptedException
>>>>  {
>>>>    p_context.getCounter(Counters.NBLINES).increment(1);
>>>>    p_context.write(one,oneL);     } // end map
>>>> } // end Mapper
>>>>
>>>> : Test :
>>>> --------
>>>>
>>>> @Test
>>>> public void testLCMapper() throws IOException, InterruptedException
>>>> {     LCMapper mapper = new LCMapper();
>>>>  Text value = new Text("ligneA\nlignB\n");
>>>>  // mock the map job execution context
>>>>  Context context = mock(Context.class);
>>>>  try {
>>>>    mapper.map(null, value, context);
>>>>  } catch (NullPointerException e)
>>>>  {
>>>>    // exception raised by context.getCounter(...) null pointer
>>>>    // HOW TO MOCK ???
>>>>    // Counter is a static enum from the englobing class ...
>>>>    // issue with variable visibility (from map function) in this test
>>>>  }
>>>>  // verify that the Mapper issued (one,oneL) pair
>>>>  verify(context).write(one,oneL);
>>>>       } // end testLCMapper
>>>>
>>>>
>>>>
>>

Mime
View raw message