hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marek Miglinski <mmiglin...@seven.com>
Subject RE: Capacity Scheduler problem
Date Thu, 12 Jan 2012 09:37:14 GMT
Thanks for the help. I've forgot about "mapred.capacity-scheduler.queue.<queue-name>.user-limit-factor"
which by default is set to 1, changed it to 10 and now it allocates resources from other queues.

Marek M.
From: Bai Shen [baishen.lists@gmail.com]
Sent: Wednesday, January 11, 2012 11:10 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Capacity Scheduler problem

Nope.  Like I said, all I did was change mapred.fairscheduler.assignmultiple to false on my
cluster and that fixed the issue.

On Wed, Jan 11, 2012 at 5:31 AM, Marek Miglinski <mmiglinski@seven.com<mailto:mmiglinski@seven.com>>

That's not the case, I've even removed hadoop-fairscheduler-0.20.2-cdh3u2.jar from hadoop
lib folder and fair-scheduler.xml from hadoop conf folder and it didn't help... Any ideas?

From: Bai Shen [baishen.lists@gmail.com<mailto:baishen.lists@gmail.com>]
Sent: Tuesday, January 10, 2012 9:35 PM
To: mapreduce-user@hadoop.apache.org<mailto:mapreduce-user@hadoop.apache.org>
Subject: Re: Capacity Scheduler problem

Turn off the fairscheduler multiple task assign setting.  I just had the same problem with
my cluster.

On Tue, Jan 10, 2012 at 11:34 AM, Marek Miglinski <mmiglinski@seven.com<mailto:mmiglinski@seven.com><mailto:mmiglinski@seven.com<mailto:mmiglinski@seven.com>>>
Hello guys,

1. I have concern with my 3 node cluster, I run capacity scheduler with 4 queues and one has
30% of cluster resources, the problem is that when I schedule a job, all tasks are assigned
to one single node which takes all of it's mappers and works quite slow. Are there any settings
for mapred/capacity scheduler to assign mappers on all nodes by even number of tasks?

2. I've set my capacity-scheduler.xml settings "mapred.capacity-scheduler.queue.job1.capacity"
for all queues as wanted but jobtracker doesn't reallocate resources from free queues if only
one is working, why?

Marek M.

View raw message