hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: How to get jobconf variables in streaming's mapper/reducer?
Date Sat, 16 May 2009 00:01:53 GMT
Hi Steve,

The variables are transformed before going to the mappers.
mapred.task.partition turns into $MAPRED_TASK_PARTITION to be more unix-y

-Todd

On Fri, May 15, 2009 at 4:52 PM, Steve Gao <steve.gao@yahoo.com> wrote:

> I am using streaming with perl, and I want to get jobconf variable values.
> As many tutorials say they are in environment, but I can not get them.
>
> For example, in reducer:
> while (<STDIN>){
>   my $part = $ENV{"mapred.task.partition"};
>   print ("$part\n");
> }
>
> It turns out that  $ENV{"mapred.task.partition"} is not defined.
>
> HOWEVER, I can get myself defined variable value. For example:
>
>  $HADOOP_HOME/bin/hadoop  \
>  jar $HADOOP_HOME/hadoop-streaming.jar \
>      -input file1 \
>      -output myOutputDir \
>      -mapper mapper \
>      -reducer reducer \
>      -jobcont arg=test
>
> In reducer:
>
> while (<STDIN>){
>
>   my $part2 = $ENV{"arg"};
>
>   print ("$part2\n");
>
> }
>
>
> It works.
>
> Anybody knows why is that? How to get jobconf variables in streaming?
> Thanks lot!
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message