hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rick Cox" <rick....@gmail.com>
Subject Re: process limits for streaming jar
Date Fri, 27 Jun 2008 16:15:46 GMT
On Fri, Jun 27, 2008 at 08:57, Chris Anderson <jchris@grabb.it> wrote:

> The problem is that when there are a large number of map tasks to
> complete, Hadoop doesn't seem to obey the map.tasks.maximum. Instead,
> it is spawning 8 map tasks per tasktracker (even when I change the
> mapred.tasktracker.map.tasks.maximum in hadoop-site.xml to 2, on the
> master). The cluster was booted with the setting at 8. Do I need to
> change hadoop-site.xml on all the slaves, and restart the task
> trackers, in order to make the limit apply? That seems unlikely - I'd
> really like to manage this parameter on a per-job level.
>

Yes, mapred.tasktracker.map.tasks.maximum is configured per
tasktracker on startup. It can't be configured per job because it's
not a job-scope parameter (if there are multiple concurrent jobs, they
have to share the task limit).

rick

Mime
View raw message