hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagat Singh <jagatsi...@gmail.com>
Subject Re: Errors about MRunit
Date Sun, 21 Apr 2013 02:39:41 GMT
Hello,

Please see this with Maven setup.

https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+Tutorial

Thanks


On Sun, Apr 21, 2013 at 12:33 PM, Rishi Yadav <rishi@infoobjects.com> wrote:

> Maven is not necessary to use hadoop or MRUnit. That being said I am a big
> fan of maven. Please find attached wordcount with maven.
>
>
>
> On Sat, Apr 20, 2013 at 7:17 PM, 姚吉龙 <geelongyao@gmail.com> wrote:
>
>> Thank for your reply
>> But I think the main problem is that I do konw how to use maven with
>> hadoop or MRunit
>> Any body can give me a example for MRunit project
>>
>>
>>
>> 2013/4/21 Rishi Yadav <rishi@infoobjects.com>
>>
>>> your problem is simple, you are mixing mapred (old api) and
>>> mapreduce(new api) libraries. MRUnit has implementation for both apis.
>>>
>>> Here's an example of WordCountTest with use of new api.
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------------------
>>>
>>> package com.infoobjects.hadoop.wc;
>>>
>>>
>>> import java.util.ArrayList;
>>>
>>> import java.util.List;
>>>
>>>
>>> import org.apache.hadoop.io.IntWritable;
>>>
>>> import org.apache.hadoop.io.LongWritable;
>>>
>>> import org.apache.hadoop.io.Text;
>>>
>>> import org.apache.hadoop.mrunit.mapreduce.MapDriver;
>>>
>>> import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
>>>
>>> import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
>>>
>>> import org.junit.Before;
>>>
>>> import org.junit.Test;
>>>
>>>
>>> public class WordCountTest {
>>>
>>> MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
>>>
>>> ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
>>>
>>> MapReduceDriver<LongWritable, Text, Text, IntWritable, Text,
>>> IntWritable> mapReduceDriver;
>>>
>>>
>>>  @Before
>>>
>>> public void init() {
>>>
>>>  WordMapper mapper = new WordMapper();
>>>
>>>  WordReducer reducer = new WordReducer();
>>>
>>>  mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
>>>
>>>  mapDriver.setMapper(mapper);
>>>
>>>     reduceDriver = ReduceDriver.newReduceDriver(reducer);
>>>
>>>     mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper,
>>> reducer);
>>>
>>>  }
>>>
>>>
>>>  @Test
>>>
>>> public void testMapper() {
>>>
>>>  mapDriver.withInput(new LongWritable(1), new Text("foo bar"));
>>>
>>>  mapDriver.withOutput(new Text("foo"), new IntWritable(1));
>>>
>>>  mapDriver.withOutput(new Text("bar"), new IntWritable(1));
>>>
>>>  mapDriver.runTest();
>>>
>>> }
>>>
>>>     @Test
>>>
>>>   public void testReducer() {
>>>
>>>     List<IntWritable> values = new ArrayList<IntWritable>();
>>>
>>>     values.add(new IntWritable(1));
>>>
>>>     values.add(new IntWritable(1));
>>>
>>>     reduceDriver.withInput(new Text("foo"), values);
>>>
>>>     reduceDriver.withOutput(new Text("foo"), new IntWritable(2));
>>>
>>>     reduceDriver.runTest();
>>>
>>>   }
>>>
>>>
>>>
>>>   @Test
>>>
>>>   public void testMapReduce() {
>>>
>>>    mapReduceDriver.withInput(new LongWritable(1), new Text("brian felix"
>>> ));
>>>
>>>    mapReduceDriver.withOutput(new Text("foo"), new IntWritable(1));
>>>
>>>    mapReduceDriver.withOutput(new Text("bar"), new IntWritable(1));
>>>
>>>    mapReduceDriver.runTest();
>>>
>>>   }
>>>
>>>
>>> }
>>>
>>> Thanks and Regards,
>>>
>>> Rishi Yadav
>>>
>>> (o) 408.988.2000x113 ||  (f) 408.716.2726
>>>
>>> InfoObjects Inc || http://www.infoobjects.com *(Big Data Solutions)*
>>>
>>> *INC 500 Fastest growing company in 2012 || 2011*
>>>
>>> *Best Place to work in Bay Area 2012 - *SF Business Times and the
>>> Silicon Valley / San Jose Business Journal
>>>
>>> 2041 Mission College Boulevard, #280 || Santa Clara, CA 95054
>>>
>>>
>>>
>>>
>>> On Sat, Apr 20, 2013 at 7:14 AM, 姚吉龙 <geelongyao@gmail.com> wrote:
>>>
>>>> This is what I got form my eclipse. Why still errors about the lib from
>>>> hadoop
>>>> [image: 内嵌图片 1][image: 内嵌图片 2]
>>>> anybody tell me how to use MRunit and Maven
>>>>
>>>>
>>>> 2013/4/20 Hemanth Yamijala <yhemanth@thoughtworks.com>
>>>>
>>>>> Hi,
>>>>>
>>>>> If your goal is to use the new API, I am able to get it to work with
>>>>> the following maven configuration:
>>>>>
>>>>>     <dependency>
>>>>>       <groupId>org.apache.mrunit</groupId>
>>>>>       <artifactId>mrunit</artifactId>
>>>>>       <version>0.9.0-incubating</version>
>>>>>       <classifier>hadoop1</classifier>
>>>>>     </dependency>
>>>>>
>>>>> If I switch with classifier hadoop2, I get the same errors as what you
>>>>> facing.
>>>>>
>>>>> Thanks
>>>>> Hemanth
>>>>>
>>>>>
>>>>> On Sat, Apr 20, 2013 at 3:42 PM, 姚吉龙 <geelongyao@gmail.com>
wrote:
>>>>>
>>>>>> Hi Everyone
>>>>>>
>>>>>> I am testing my MR programe with MRunit, it's version
>>>>>> is mrunit-0.9.0-incubating-hadoop2. My hadoop version is 1.0.4
>>>>>> The error trace is below:
>>>>>>
>>>>>> java.lang.IncompatibleClassChangeError: Found class
>>>>>> org.apache.hadoop.mapreduce.TaskInputOutputContext, but interface
was
>>>>>> expected
>>>>>> at
>>>>>> org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper.createCommon(MockContextWrapper.java:53)
>>>>>>  at
>>>>>> org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.create(MockMapContextWrapper.java:70)
>>>>>> at
>>>>>> org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.<init>(MockMapContextWrapper.java:62)
>>>>>>  at
>>>>>> org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:217)
>>>>>> at
>>>>>> org.apache.hadoop.mrunit.MapDriverBase.runTest(MapDriverBase.java:150)
>>>>>>  at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:137)
>>>>>> at UnitTest.testMapper(UnitTest.java:41)
>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>  at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>  at junit.framework.TestCase.runTest(TestCase.java:168)
>>>>>> at junit.framework.TestCase.runBare(TestCase.java:134)
>>>>>>  at junit.framework.TestResult$1.protect(TestResult.java:110)
>>>>>> at junit.framework.TestResult.runProtected(TestResult.java:128)
>>>>>>  at junit.framework.TestResult.run(TestResult.java:113)
>>>>>> at junit.framework.TestCase.run(TestCase.java:124)
>>>>>>  at junit.framework.TestSuite.runTest(TestSuite.java:232)
>>>>>> at junit.framework.TestSuite.run(TestSuite.java:227)
>>>>>>  at
>>>>>> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79)
>>>>>> at
>>>>>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>>>>>>  at
>>>>>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>>>>>> at
>>>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>>>>>>  at
>>>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>>>>>> at
>>>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>>>>>>  at
>>>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>>>>>
>>>>>>
>>>>>> Anyone has idea?
>>>>>>
>>>>>> BRs
>>>>>> Geelong
>>>>>>
>>>>>> --
>>>>>> From Good To Great
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> From Good To Great
>>>>
>>>
>>>
>>
>>
>> --
>> From Good To Great
>>
>
>

Mime
View raw message