hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miles Osborne <mi...@inf.ed.ac.uk>
Subject Re: mapred.tasktracker.map.tasks.maximum
Date Wed, 22 Apr 2009 00:28:30 GMT
they are the places to check.  a job can itself over-ride the number
of mappers and reducers.  for example,  using streaming, i often state
the number of mappers and reducers i want to use:

-jobconf mapred.reduce.tasks=30

this would tell hadoop to use 30 reducers, for example.

if you don't have enough memory to run a mapper then you will see
error messages logged somewhere.  perhaps it might be useful to check
all the logs


2009/4/22 javateck javateck <javateck@gmail.com>:
> I want to have something to clarify, for the max task slots, are these
> places to check:
> 1. hadoop-site.xml
> 2. the specific job's job.conf which can be retrieved though the job, for
> example, logs/job_200904212336_0002_conf.xml
> Any other place to limit the task map counts?
> In my case, it's strange, because I set 10 for "
> mapred.tasktracker.map.tasks.maximum", and I check the job's conf is also
> 10, but actual hadoop is just using 2 map jobs.
> On Tue, Apr 21, 2009 at 1:20 PM, javateck javateck <javateck@gmail.com>wrote:
>> I set my "mapred.tasktracker.map.tasks.maximum" to 10, but when I run a
>> task, it's only using 2 out of 10, any way to know why it's only using 2?
>> thanks

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

View raw message