Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E015291AF for ; Fri, 30 Dec 2011 14:23:11 +0000 (UTC) Received: (qmail 85175 invoked by uid 500); 30 Dec 2011 14:23:10 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 85133 invoked by uid 500); 30 Dec 2011 14:23:10 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 85125 invoked by uid 99); 30 Dec 2011 14:23:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Dec 2011 14:23:10 +0000 X-ASF-Spam-Status: No, hits=1.6 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of egolan74@gmail.com designates 209.85.160.48 as permitted sender) Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Dec 2011 14:23:04 +0000 Received: by pbaa12 with SMTP id a12so8977197pba.35 for ; Fri, 30 Dec 2011 06:22:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=5Zai0rbeO0SouziHDnc8SipJ02235H70rgdcIlIEXW8=; b=qeZBcI4gBUyS//Pkcqn32AOZTl7sjwDmuZJB0mJ3ff9uHBWD6/bhd210oT0/uj+HSb CO+xfbfY8HA2PAlERF1PNXkfpmdgJOhyb0O3MW1Kep0hBnWM32eetS5Jvop7A7W85DaL b5o23ba+hM2QfX4Tqc9DI2iCPSbT2VVDPIOvs= Received: by 10.68.74.230 with SMTP id x6mr94823714pbv.49.1325254963189; Fri, 30 Dec 2011 06:22:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.68.48.166 with HTTP; Fri, 30 Dec 2011 06:22:22 -0800 (PST) In-Reply-To: <2778E4CD-DBA8-4DCA-BC13-BE1F4B459E14@cloudera.com> References: <2778E4CD-DBA8-4DCA-BC13-BE1F4B459E14@cloudera.com> From: Eyal Golan Date: Fri, 30 Dec 2011 16:22:22 +0200 Message-ID: Subject: Re: instantiation of classes in MR To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d041390ef5157ff04b54ff7f1 X-Virus-Checked: Checked by ClamAV on apache.org --f46d041390ef5157ff04b54ff7f1 Content-Type: text/plain; charset=ISO-8859-1 Great News !! Thanks for the info. So using reflection, I can inject different implementations of interfaces (services) for the mapper (or reducer). And this way I can test a mapper (or reducer). Just by reflecting a stub instead of a real implementation. Thanks, Eyal Golan egolan74@gmail.com Visit: http://jvdrums.sourceforge.net/ LinkedIn: http://www.linkedin.com/in/egolan74 Skype: egolan74 P Save a tree. Please don't print this e-mail unless it's really necessary On Fri, Dec 30, 2011 at 2:50 PM, Harsh J wrote: > Eyal, > > Yes, it is right to think of each Task attempt being one individual JVM > running individually on any added Node. Multiple slots would mean multiple > VMs in parallel as well. Yes, your use of reflection to build your objects > will work just fine -- its all user-side java code that is executed. > > On 30-Dec-2011, at 4:42 PM, Eyal Golan wrote: > > Hi, > > I want to understand a basic concept in MR. > > If a mapper creates an instance of some class (using the 'new' operator), > then the created class exists ONCE in the VM of this node. > For each node. > Correct? > > Now, > what if instead of using the 'new' operator, the class is created using > reflection. > Is it valid in a MR? > Will only one instance of the created class be existing in that node? > > Thanks, > > > Eyal > > Eyal Golan > egolan74@gmail.com > > Visit: http://jvdrums.sourceforge.net/ > LinkedIn: http://www.linkedin.com/in/egolan74 > Skype: egolan74 > > P Save a tree. Please don't print this e-mail unless it's really necessary > > > --f46d041390ef5157ff04b54ff7f1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Great News !!
Thanks for the info.

So using reflection, I can inject different implementations of interf= aces (services) for the mapper (or reducer).
And this way I can t= est a mapper (or reducer).
Just by reflecting a stub instead of a real implementation.
=
Thanks,






On Fri, Dec 30, 2011 at 2:50 PM, Harsh J= <harsh@cloudera= .com> wrote:
Eyal,

Yes, it is rig= ht to think of each Task attempt being one individual JVM running individua= lly on any added Node. Multiple slots would mean multiple VMs in parallel a= s well. Yes, your use of reflection to build your objects will work just fi= ne -- its all user-side java code that is executed.

On 30-Dec-2011, at 4:42 PM, Eyal = Golan wrote:

Hi,
I want to understand a basic concept in MR.

If a mapper creates an instance of some class (using the 'new'= operator), then the created class exists ONCE in the VM of this node.
For each node.
Correct?

Now,
<= div>what if instead of using the 'new' operator, the class is creat= ed using reflection.
Is it valid in a MR?
Will only one= instance of the created class be existing in that node?

Thanks,


Eyal

Eyal Golan
egolan74@gmail.com

Visit: = http://jvdrum= s.sourceforge.net/
LinkedIn: http://www.linkedin.com/in/egolan74
Skype: egolan74

P=A0 Sa= ve a tree. Please don't print this e-mail unless it's really necess= ary



--f46d041390ef5157ff04b54ff7f1--