hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 姚吉龙 <geelong...@gmail.com>
Subject Re: Errors about MRunit
Date Sun, 21 Apr 2013 02:17:53 GMT
Thank for your reply
But I think the main problem is that I do konw how to use maven with hadoop
or MRunit
Any body can give me a example for MRunit project



2013/4/21 Rishi Yadav <rishi@infoobjects.com>

> your problem is simple, you are mixing mapred (old api) and mapreduce(new
> api) libraries. MRUnit has implementation for both apis.
>
> Here's an example of WordCountTest with use of new api.
>
>
> ----------------------------------------------------------------------------------------------------------------------
>
> package com.infoobjects.hadoop.wc;
>
>
> import java.util.ArrayList;
>
> import java.util.List;
>
>
> import org.apache.hadoop.io.IntWritable;
>
> import org.apache.hadoop.io.LongWritable;
>
> import org.apache.hadoop.io.Text;
>
> import org.apache.hadoop.mrunit.mapreduce.MapDriver;
>
> import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
>
> import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
>
> import org.junit.Before;
>
> import org.junit.Test;
>
>
> public class WordCountTest {
>
> MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
>
> ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
>
> MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>
> mapReduceDriver;
>
>
>  @Before
>
> public void init() {
>
>  WordMapper mapper = new WordMapper();
>
>  WordReducer reducer = new WordReducer();
>
>  mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
>
>  mapDriver.setMapper(mapper);
>
>     reduceDriver = ReduceDriver.newReduceDriver(reducer);
>
>     mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);
>
>  }
>
>
>  @Test
>
> public void testMapper() {
>
>  mapDriver.withInput(new LongWritable(1), new Text("foo bar"));
>
>  mapDriver.withOutput(new Text("foo"), new IntWritable(1));
>
>  mapDriver.withOutput(new Text("bar"), new IntWritable(1));
>
>  mapDriver.runTest();
>
> }
>
>     @Test
>
>   public void testReducer() {
>
>     List<IntWritable> values = new ArrayList<IntWritable>();
>
>     values.add(new IntWritable(1));
>
>     values.add(new IntWritable(1));
>
>     reduceDriver.withInput(new Text("foo"), values);
>
>     reduceDriver.withOutput(new Text("foo"), new IntWritable(2));
>
>     reduceDriver.runTest();
>
>   }
>
>
>
>   @Test
>
>   public void testMapReduce() {
>
>    mapReduceDriver.withInput(new LongWritable(1), new Text("brian felix"
> ));
>
>    mapReduceDriver.withOutput(new Text("foo"), new IntWritable(1));
>
>    mapReduceDriver.withOutput(new Text("bar"), new IntWritable(1));
>
>    mapReduceDriver.runTest();
>
>   }
>
>
> }
>
> Thanks and Regards,
>
> Rishi Yadav
>
> (o) 408.988.2000x113 ||  (f) 408.716.2726
>
> InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
>
> *INC 500 Fastest growing company in 2012 || 2011*
>
> *Best Place to work in Bay Area 2012 - *SF Business Times and the Silicon
> Valley / San Jose Business Journal
>
> 2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
>
>
>
>
> On Sat, Apr 20, 2013 at 7:14 AM, 姚吉龙 <geelongyao@gmail.com> wrote:
>
>> This is what I got form my eclipse. Why still errors about the lib from
>> hadoop
>> [image: 内嵌图片 1][image: 内嵌图片 2]
>> anybody tell me how to use MRunit and Maven
>>
>>
>> 2013/4/20 Hemanth Yamijala <yhemanth@thoughtworks.com>
>>
>>> Hi,
>>>
>>> If your goal is to use the new API, I am able to get it to work with the
>>> following maven configuration:
>>>
>>>     <dependency>
>>>       <groupId>org.apache.mrunit</groupId>
>>>       <artifactId>mrunit</artifactId>
>>>       <version>0.9.0-incubating</version>
>>>       <classifier>hadoop1</classifier>
>>>     </dependency>
>>>
>>> If I switch with classifier hadoop2, I get the same errors as what you
>>> facing.
>>>
>>> Thanks
>>> Hemanth
>>>
>>>
>>> On Sat, Apr 20, 2013 at 3:42 PM, 姚吉龙 <geelongyao@gmail.com> wrote:
>>>
>>>> Hi Everyone
>>>>
>>>> I am testing my MR programe with MRunit, it's version
>>>> is mrunit-0.9.0-incubating-hadoop2. My hadoop version is 1.0.4
>>>> The error trace is below:
>>>>
>>>> java.lang.IncompatibleClassChangeError: Found class
>>>> org.apache.hadoop.mapreduce.TaskInputOutputContext, but interface was
>>>> expected
>>>> at
>>>> org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper.createCommon(MockContextWrapper.java:53)
>>>>  at
>>>> org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.create(MockMapContextWrapper.java:70)
>>>> at
>>>> org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.<init>(MockMapContextWrapper.java:62)
>>>>  at
>>>> org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:217)
>>>> at
>>>> org.apache.hadoop.mrunit.MapDriverBase.runTest(MapDriverBase.java:150)
>>>>  at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:137)
>>>> at UnitTest.testMapper(UnitTest.java:41)
>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>  at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>  at junit.framework.TestCase.runTest(TestCase.java:168)
>>>> at junit.framework.TestCase.runBare(TestCase.java:134)
>>>>  at junit.framework.TestResult$1.protect(TestResult.java:110)
>>>> at junit.framework.TestResult.runProtected(TestResult.java:128)
>>>>  at junit.framework.TestResult.run(TestResult.java:113)
>>>> at junit.framework.TestCase.run(TestCase.java:124)
>>>>  at junit.framework.TestSuite.runTest(TestSuite.java:232)
>>>> at junit.framework.TestSuite.run(TestSuite.java:227)
>>>>  at
>>>> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
>>>> at
>>>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>>>>  at
>>>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>>>>  at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>>>>  at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>>>
>>>>
>>>> Anyone has idea?
>>>>
>>>> BRs
>>>> Geelong
>>>>
>>>> --
>>>> From Good To Great
>>>>
>>>
>>>
>>
>>
>> --
>> From Good To Great
>>
>
>


-- 
>From Good To Great

Mime
View raw message