hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Gao <steve....@yahoo.com>
Subject How to get jobconf variables in streaming's mapper/reducer?
Date Fri, 15 May 2009 23:52:52 GMT
I am using streaming with perl, and I want to get jobconf variable values. As many tutorials
say they are in environment, but I can not get them. 

For example, in reducer:
while (<STDIN>){
  my $part = $ENV{"mapred.task.partition"};
  print ("$part\n");
}

It turns out that  $ENV{"mapred.task.partition"} is not defined.

HOWEVER, I can get myself defined variable value. For example:

 $HADOOP_HOME/bin/hadoop  \
 jar $HADOOP_HOME/hadoop-streaming.jar \
     -input file1 \
     -output myOutputDir \
     -mapper mapper \
     -reducer reducer \
     -jobcont arg=test

In reducer:

while (<STDIN>){

  my $part2 = $ENV{"arg"};

  print ("$part2\n");

}


It works.

Anybody knows why is that? How to get jobconf variables in streaming? Thanks lot!



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message