hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: Please help with hadoop configuration parameter set and get
Date Fri, 17 Dec 2010 17:32:18 GMT
Statics won't work the way you might think because different mappers and
different reducers are all running in different JVM's.  It might work in
local mode, but don't kid yourself about it working in a distributed mode.
 It won't.

On Fri, Dec 17, 2010 at 8:31 AM, Peng, Wei <Wei.Peng@xerox.com> wrote:

> Arindam, how to set this global static Boolean variable?
> I have tried to do something similarly yesterday in the following:
> Public class BFSearch
> {
>        Private static boolean expansion;
>        Public static class MapperClass {if no nodes expansion = false;}
>        Public static class ReducerClass
>        Public static void main {expansion = true; run job;
> print(expansion)}
> }
> In this case, expansion is still true.
> I will look at hadoop counter and report back here later.
>
> Thank you for all your help
> Wei
>
> -----Original Message-----
> From: Arindam Khaled [mailto:akhaled@utdallas.edu]
> Sent: Friday, December 17, 2010 10:35 AM
> To: common-user@hadoop.apache.org
> Subject: Re: Please help with hadoop configuration parameter set and get
>
> I did something like this using a global static boolean variable
> (flag) while I was implementing breadth first IDA*. In my case, I set
> the flag to something else if a solution was found, which was examined
> in the reducer.
>
> I guess in your case, since you know that if the mappers don't produce
> anything the reducers won't have anything as input, if I am not wrong.
>
> And I had chaining map-reduce jobs (
> http://developer.yahoo.com/hadoop/tutorial/module4.html
>  ) running until a solution was found.
>
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>
> > Hi,
> >
> >
> >
> > I am a newbie of hadoop.
> >
> > Today I was struggling with a hadoop problem for several hours.
> >
> >
> >
> > I initialize a parameter by setting job configuration in main.
> >
> > E.g. Configuration con = new Configuration();
> >
> > con.set("test", "1");
> >
> > Job job = new Job(con);
> >
> >
> >
> > Then in the mapper class, I want to set "test" to "2". I did it by
> >
> > context.getConfiguration().set("test","2");
> >
> >
> >
> > Finally in the main method, after the job is finished, I check the
> > "test" again by
> >
> > job.getConfiguration().get("test");
> >
> >
> >
> > However, the value of "test" is still "1".
> >
> >
> >
> > The reason why I want to change the parameter inside Mapper class is
> > that I want to determine when to stop an iteration in the main method.
> > For example, for doing breadth-first search, when there is no new
> > nodes
> > are added for further expansion, the searching iteration should stop.
> >
> >
> >
> > Your help will be deeply appreciated. Thank you
> >
> >
> >
> > Wei
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message