hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "TaskExecutionEnvironment" by AndyPavlo
Date Fri, 16 May 2008 17:04:47 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by AndyPavlo:
http://wiki.apache.org/hadoop/TaskExecutionEnvironment

The comment on the change is:
Added information about how to get the JobConf inside of a map function

------------------------------------------------------------------------------
  || map.input.length || long || The number of bytes in the map input split ||
  || mapred.work.output.dir || String || The task's temporary output directory ||
  
+ == Accessing JobConf in MapReduce Programs ==
+ 
+ The following is an example of how to access the current JobConf from inside of your Map
or Reduce functions while they are executing. One key thing to note is that the Map class
must not be static (as they are in the examples) and that it needs to override {{{configure}}}
in order to get access to the JobConf. You will need to set the property before you call {{{JobClient.runJob()}}}.
This same concept can be used for Reducer and Combiner class implementations.
+ 
+ {{{
+ import java.io.IOException;
+ import org.apache.hadoop.io.*;
+ import org.apache.hadoop.mapred.*;
+ 
+ public class Map extends MapReduceBase implements Mapper<Text, Text, Text, IntWritable>
{
+    protected Integer MIN_VALUE = null;
+    protected static final String MIN_VALUE_KEY = "test.minvalue";
+ 
+    // Set the MIN_VALUE property
+    public void configure(JobConf job) {
+       super.configure(job);
+       // Get the min value from the current JobConf object
+       // If it was not set, then the resulting value will be null
+       String property = job.get(Map.MIN_VALUE_KEY);
+       if (property == null) {
+          System.err.println("ERROR: The property '" + this.MIN_VALUE_KEY + "' was not set");
+          System.exit(1);
+       }
+       this.MIN_VALUE = Integer.parseInt(property);
+    }
+ 
+    // Check whether the value is greater than our MIN_VALUE
+    public void map(Text key, Text value, OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
+       Integer temp = Integer.valueOf(value.toString());
+       if (temp > this.MIN_VALUE) {
+          output.collect(key, new IntWritable(temp));
+       }
+       return;
+    }
+ } // END CLASS
+ }}}
+ 

Mime
View raw message