hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rasit OZDAS <rasitoz...@gmail.com>
Subject Re: Overriding mapred.tasktracker.map.tasks.maximum with -jobconf
Date Wed, 18 Feb 2009 15:14:28 GMT
John, did you try -D option instead of -jobconf,

I had -D option in my code, I changed it with -jobconf, this is what I get:

  -input    <path>     DFS input file(s) for the Map step
  -output   <path>     DFS output directory for the Reduce step
  -mapper   <cmd|JavaClassName>      The streaming command to run
  -combiner <JavaClassName> Combiner has to be a Java class
  -reducer  <cmd|JavaClassName>      The streaming command to run
  -file     <file>     File/dir to be shipped in the Job jar file
  -outputformat TextOutputFormat(default)|JavaClassName  Optional.
  -partitioner JavaClassName  Optional.
  -numReduceTasks <num>  Optional.
  -inputreader <spec>  Optional.
  -cmdenv   <n>=<v>    Optional. Pass env.var to streaming commands
  -mapdebug <path>  Optional. To run this script when a map task fails
  -reducedebug <path>  Optional. To run this script when a reduce task fails


Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|jobtracker:port>    specify a job tracker
-files <comma separated list of files>    specify comma separated files to
be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files
to include in the classpath.
-archives <comma separated list of archives>    specify comma separated
archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

For more details about these options:
Use $HADOOP_HOME/bin/hadoop jar build/hadoop-streaming.jar -info

I think -jobconf is not used in v.0.19 .

2009/2/18 S D <sd.codewarrior@gmail.com>

> I'm having trouble overriding the maximum number of map tasks that run on a
> given machine in my cluster. The default value of
> mapred.tasktracker.map.tasks.maximum is set to 2 in hadoop-default.xml.
> When
> running my job I passed
> -jobconf mapred.tasktracker.map.tasks.maximum=1
> to limit map tasks to one per machine but each machine was still allocated
> 2
> map tasks (simultaneously).  The only way I was able to guarantee a maximum
> of one map task per machine was to change the value of the property in
> hadoop-site.xml. This is unsatisfactory since I'll often be changing the
> maximum on a per job basis. Any hints?
> On a different note, when I attempt to pass params via -D I get a usage
> message; when I use -jobconf the command goes through (and works in the
> case
> of mapred.reduce.tasks=0 for example) but I get  a deprecation warning).
> Thanks,
> John

M. Raşit ÖZDAŞ

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message