Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=aQEg/xYEtFaJh2Qfnkqias0IOUZTdLYWHw/htvWEF1VU3dYqbwAWdcEMaWkXdWKo
Message-Id: <BCCA78A2-183B-491F-A74D-B5A3ED317D93@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <80576ef0911112228k24322a9dn9511ae55bf379dfa@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-7-533592655
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Can I wire a mapper and reducer class?
Date: Thu, 12 Nov 2009 00:14:26 -0800
References: <80576ef0911112228k24322a9dn9511ae55bf379dfa@mail.gmail.com>


--Apple-Mail-7-533592655
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

You can't do that, at least not yet: https://issues.apache.org/jira/browse/MAPREDUCE-1183

Arun

On Nov 11, 2009, at 10:28 PM, Java Developer wrote:

> I want to know if I can "wire" mapper and reducer classes. By "wire"  
> I mean to use inversion of control (or dependency injection). For  
> example, I see in a lot of books and on the Map/Reduce tutorial page  
> (http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html),  
> something like this:
>
> conf.setMapperClass(MyMapper.class);
> conf.setReducerClass(MyReducer.class);
>
> However, I want do something like this:
>
> MyMapper mapper = new MyMapper();
> mapper.setSomeProperty("something");
>
> MyReducer reducer = new MyReducer();
> reducer.setSomeProperty("somethingElse");
>
> JobConf conf = new JobConf();
> conf.setMapper(mapper);
> conf.setReducer(reducer);
>
> Is this possible? I'm still learning MapReduce so I don't know if  
> this is possible. But all examples that I have come across follow  
> the pattern on the tutorial page. There is a work around that I've  
> been forced to. In my mapper and reducer class, I can grab some  
> properties file and then fill in the values that I need (for example  
> for JDBC connection). I see the problem that when I run this on a  
> distributed environment, I'll need the properties files there as  
> well. Does the mapper and reducer classes have a reference back to  
> the JobConf object? If so, I suppose I can get the properties there.
>
> Thanks.


--Apple-Mail-7-533592655
Content-Type: text/html;
	charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

<html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; ">You can't do that, at least not =
yet:&nbsp;<a =
href=3D"https://issues.apache.org/jira/browse/MAPREDUCE-1183">https://issu=
es.apache.org/jira/browse/MAPREDUCE-1183</a><div><br></div><div>Arun</div>=
<div><br><div><div>On Nov 11, 2009, at 10:28 PM, Java Developer =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite">I want to know if I can "wire" mapper and reducer classes. =
By "wire" I mean to use inversion of control (or dependency injection). =
For example, I see in a lot of books and on the Map/Reduce tutorial page =
(<a =
href=3D"http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html"=
>http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html</a>), =
something like this:<br> =
<br>conf.setMapperClass(MyMapper.class);<br>conf.setReducerClass(MyReducer=
.class);<br><br>However, I want do something like this:<br><br>MyMapper =
mapper =3D new MyMapper();<br>mapper.setSomeProperty("something");<br> =
<br>MyReducer reducer =3D new =
MyReducer();<br>reducer.setSomeProperty("somethingElse");<br><br>JobConf =
conf =3D new =
JobConf();<br>conf.setMapper(mapper);<br>conf.setReducer(reducer);<br><br>=
Is this possible? I'm still learning MapReduce so I don't know if this =
is possible. But all examples that I have come across follow the pattern =
on the tutorial page. There is a work around that I've been forced to. =
In my mapper and reducer class, I can grab some properties file and then =
fill in the values that I need (for example for JDBC connection). I see =
the problem that when I run this on a distributed environment, I'll need =
the properties files there as well. Does the mapper and reducer classes =
have a reference back to the JobConf object? If so, I suppose I can get =
the properties there. <br> =
<br>Thanks.<br></blockquote></div><br></div></body></html>=

--Apple-Mail-7-533592655--