hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: non-static map or reduce classes?
Date Tue, 27 May 2008 17:50:13 GMT

That sounds just right.


On 5/26/08 7:05 AM, "Deyaa Adranale" <deyaa.adranale@iais.fraunhofer.de>
wrote:

> since the information needed in the MapReduce is just small data as
> strings or integers, i did not prefer to write it to the file system.
> otherwise, i used the methods conf.setInt(name,intvalue) and
> conf.set(name,stringvalue) to give the data to the JobConf object. Then
> inside mapper.configure(JobConf) or reducer.configure(JobConf), i used:
> job.getInt(name,defaultvalue) and job.get(name) to get the values of the
> data assigned previously.
> 
> i think if i had large amounts of data to be shared I could store it in
> the DFS and add it to the configeration by calling:
> conf.addResource(...) and read it in a similar way inside the
> mapper/reduce.configure method.
> 
> is what i said right? any corrections?
> 
> thanks for the support
> 
> Deyaa
> 
> 
> 
> 
> Deyaa Adranale schrieb:
>> thanks for the answer. ur suggestion is interesting to try.
>> 
>> i will try it out and write the result here.
>> 
>> 
>> 
>> Ted Dunning schrieb:
>>> The mapper and reducers are not executing on the same machine as your
>>> main
>>> program.
>>> 
>>> That means that they cannot access the data you are talking about.  The
>>> requirement that they be static classes is a reflection of this.
>>> 
>>> If you need to share data, then you need to write it to the file
>>> system and
>>> read it in the configure method on the mapper or reducer.
>>> 
>>> 
>>> On 5/21/08 3:17 AM, "Deyaa Adranale" <deyaa.adranale@iais.fraunhofer.de>
>>> wrote:
>>> 
>>>  
>>>> hello.
>>>> why do we have to set the map and reduce classes as static?
>>>> i need inside them to access some data which is not static. what i
>>>> should do?
>>>> 
>>>> non static map or reduce classes generates the following exception:
>>>> 
>>>> java.lang.RuntimeException: java.lang.NoSuchMethodException:
>>>> hadoop.examples.Simple$MyMapper.()
>>>>     at 
>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:45)
>>>> 
>>>>     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
>>>>     at 
>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:53)
>>>> 
>>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:209)
>>>>     at 
>>>> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1210)
>>>> Caused by: java.lang.NoSuchMethodException:
>>>> hadoop.examples.Simple$MyMapper.()
>>>>     at java.lang.Class.getConstructor0(Class.java:2705)
>>>>     at java.lang.Class.getDeclaredConstructor(Class.java:1984)
>>>>     at 
>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:41)
>>>> 
>>>>     ... 4 more
>>>> 
>>>> 
>>>> 
>>>> 
>>>> any ideas?
>>>> 
>>>> thanks in advance,
>>>> 
>>>> Deyaa
>>>>     
>>> 
>>> 
>>>   
>> 
>> 
> 


Mime
View raw message