hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anirudh <techie.anir...@gmail.com>
Subject Re: instantiation of classes in MR
Date Sat, 31 Dec 2011 04:50:13 GMT
Where are you creating this new class. If it is in the map function, then
it will be create a new object for each record in the split.

Also you may need to see how the JVM reuse option works. I am not too sure
of this and you may want to look at the code. If the option for JVM reuse
is set, then my understanding is for every task, a new Map task would be
created and in that case the "new" operator will create another instance
even if this statement is not in the map function.

On Fri, Dec 30, 2011 at 6:22 AM, Eyal Golan <egolan74@gmail.com> wrote:

> Great News !!
> Thanks for the info.
>
> So using reflection, I can inject different implementations of interfaces
> (services) for the mapper (or reducer).
> And this way I can test a mapper (or reducer).
> Just by reflecting a stub instead of a real implementation.
>
> Thanks,
>
>
>
> Eyal Golan
> egolan74@gmail.com
>
> Visit: http://jvdrums.sourceforge.net/
> LinkedIn: http://www.linkedin.com/in/egolan74
> Skype: egolan74
>
> P  Save a tree. Please don't print this e-mail unless it's really necessary
>
>
>
> On Fri, Dec 30, 2011 at 2:50 PM, Harsh J <harsh@cloudera.com> wrote:
>
>> Eyal,
>>
>> Yes, it is right to think of each Task attempt being one individual JVM
>> running individually on any added Node. Multiple slots would mean multiple
>> VMs in parallel as well. Yes, your use of reflection to build your objects
>> will work just fine -- its all user-side java code that is executed.
>>
>> On 30-Dec-2011, at 4:42 PM, Eyal Golan wrote:
>>
>> Hi,
>>
>> I want to understand a basic concept in MR.
>>
>> If a mapper creates an instance of some class (using the 'new' operator),
>> then the created class exists ONCE in the VM of this node.
>> For each node.
>> Correct?
>>
>> Now,
>> what if instead of using the 'new' operator, the class is created using
>> reflection.
>> Is it valid in a MR?
>> Will only one instance of the created class be existing in that node?
>>
>> Thanks,
>>
>>
>> Eyal
>>
>> Eyal Golan
>> egolan74@gmail.com
>>
>> Visit: http://jvdrums.sourceforge.net/
>> LinkedIn: http://www.linkedin.com/in/egolan74
>> Skype: egolan74
>>
>> P  Save a tree. Please don't print this e-mail unless it's really
>> necessary
>>
>>
>>
>

Mime
View raw message