Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of shrijeet@rocketfuelinc.com
 designates 209.85.161.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1C091122-8196-42CF-A0D6-7A8B78B7D115@gmail.com>
References: <1C091122-8196-42CF-A0D6-7A8B78B7D115@gmail.com>
Date: Mon, 13 Dec 2010 13:27:50 -0800
Message-ID: <AANLkTikyOM22S8NVSn_r578=qKOBn+2kc=ACcLgwxM_P@mail.gmail.com>
Subject: Re: Passing configuration options to the Map class
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf30433e3a44d29d0497516009

--20cf30433e3a44d29d0497516009
Content-Type: text/plain; charset=ISO-8859-1

Hi,

configure(JobConf conf)method in
http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/MapReduceBase.html
is your friend.

Try something like this,


public class BobsMapper extends MapReduceBase implements
Mapper<LongWritable, Text, Text, Text> {
    private Path out_path = null;
    /**
     * Use to configure map job.
     * Parameters from configure XML can be accessed here.
     * This will be called once before framework start calling map()
iteratively for given input file
     */
    public void configure(JobConf conf) {
        try {
            super.configure(conf);
            //get parameter values from configuration XML
            out_path = new Path(conf.get("outputPath"));
        }
        catch (Exception e) {
            _log.error("Problem configuring map job. " + e.getMessage());
        }
    }

    /** the mapper should not be able to access out_path.
    public void map(...) {
    }
}

On Mon, Dec 13, 2010 at 12:54 PM, Eric <eric.xkcd@gmail.com> wrote:
>
> Hi,
>
> I'm looking for a way to pass options to my Map class. More specifically,
I want my mappers to write to sequence files in a path that I can specify on
the command line. I tried setting a variable with:
>
> Jobconf conf = ........;
> conf.set("outputPath", "/foo/bar");
>
> (I'm using the deprecated libraries)
>
> I can not retrieve this setting in my mapper however. Is there another
way, or am I doing something wrong?
>
> Best regards,
> Eric

--20cf30433e3a44d29d0497516009
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div>Hi,=A0</div><div><br></div>configure(JobConf conf)method in=A0<a href=
=3D"http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapr=
ed/MapReduceBase.html">http://hadoop.apache.org/common/docs/r0.20.0/api/org=
/apache/hadoop/mapred/MapReduceBase.html</a><br>
is your=A0friend.=A0<br><br>Try something like this,=A0<div><br></div><div>=
<br>public class BobsMapper extends MapReduceBase implements Mapper&lt;Long=
Writable, Text, Text, Text&gt;=A0{<br>=A0=A0 =A0private Path out_path =3D n=
ull;<br>=A0=A0 =A0/**<br>
=A0=A0 =A0 * Use to configure map job.<br>=A0=A0 =A0 * Parameters from conf=
igure XML can be accessed here.<br>=A0=A0 =A0 * This will be called once be=
fore framework start calling map() iteratively for given input file<br>=A0=
=A0 =A0 */<br>=A0=A0 =A0public void configure(JobConf conf)=A0{<br>
=A0=A0 =A0 =A0 =A0try=A0{<br>=A0=A0 =A0 =A0 =A0 =A0 =A0super.configure(conf=
);<br>=A0=A0 =A0 =A0 =A0 =A0 =A0//get parameter values from configuration X=
ML<br>=A0=A0 =A0 =A0 =A0=A0=A0=A0 out_path =3D new Path(conf.get(&quot;outp=
utPath&quot;));<br>=A0=A0 =A0 =A0 =A0}<br>=A0=A0 =A0 =A0 =A0catch (Exceptio=
n e)=A0{<br>
=A0=A0 =A0 =A0 =A0 =A0 =A0_log.error(&quot;Problem configuring map job. &qu=
ot; + e.getMessage());<br>=A0=A0 =A0 =A0 =A0}<br>=A0=A0 =A0}<br>=A0=A0<br>=
=A0=A0 =A0/** the mapper should not be able to access=A0out_path.<br>=A0=A0=
 =A0public void map(...) {<br>=A0=A0 =A0}<br>}<br><br>
On Mon, Dec 13, 2010 at 12:54 PM, Eric &lt;<a href=3D"mailto:eric.xkcd@gmai=
l.com">eric.xkcd@gmail.com</a>&gt; wrote:<br>&gt;<br>&gt; Hi,<br>&gt;<br>&g=
t; I&#39;m looking for a way to pass options to my Map class. More specific=
ally, I want my mappers to write to sequence files in a path that I can spe=
cify on the command line. I tried setting a variable with:<br>
&gt;<br>&gt; Jobconf conf =3D ........;<br>&gt; conf.set(&quot;outputPath&q=
uot;, &quot;/foo/bar&quot;);<br>&gt;<br>&gt; (I&#39;m using the deprecated =
libraries)<br>&gt;<br>&gt; I can not retrieve this setting in my mapper how=
ever. Is there another way, or am I doing something wrong?<br>
&gt;<br>&gt; Best regards,<br>&gt; Eric<br></div>

--20cf30433e3a44d29d0497516009--