hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy <snickerdoodl...@gmail.com>
Subject pseudo-global variable constuction
Date Tue, 19 Aug 2008 20:56:57 GMT
Hello,


My M/R program is going smoothly, except for one small problem. I have a
"global" variable that is set by the user (and thus in the main function),
that I want one of my reduce functions to access. This is a read-only
variable. After some reading in the forums, I tried something like this:

file: MyGlobalVars.java
package org.apache.hadoop.examples;
public class MyGlobalVars {
    static public int nTax;
}
------

file: myLittleMRProgram.java
package.org.apache.hadoop.examples;
map function() {
   System.out.println("in map function, nTax is: " + MyGlobalVars.nTax);
}
....
main() {
MyGlobalVars.nTax = other_args.get(2);
System.out.println("in main function, nTax is: " + MyGlobalVars.nTax);
....
JobClient.runJob(conf);
....
return 0;
}
--------

When I run it, I get:
in main function, nTax is 20 (which is what I want)
in map function, nTax is 0 (<--- this is not right).


I am a little confused on how to resolve this. I apologize in advance if
this is an blatant java error; I only began programming in the language a
few weeks ago.

Since Map Reduce tries to avoid the whole shared-memory scene, I am more
than willing to have each reduce function receive a local copy of this user
defined value. However, I am a little confused on what the best way to do
this would be. As I see it, my options are:

1.) write the user defined value to the hdfs in the main function, and have
it read from the hdfs in the reduce function. I can't quite figure out the
code to this though. I know how to specify -an- input file for the map
reduce task, but if I did it this way, won't I need to specify two separate
input files?

2. Put it in the construction of the reduce object (I saw this mentioned in
the archives). How would I accomplish this exactly when the value is user
defined? Parameter Passing? If so, won't this require me changing the
underlying map reduce base (which makes me a touch nervous, since i'm still
very new to hadoop).

What would be the easiest way to do this?

Thanks in advance for the help. I appreciate your time.

-SM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message