hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manish Katyal <manish.kat...@gmail.com>
Subject Regarding Capacity Scheduler
Date Wed, 13 May 2009 16:48:09 GMT
I'm experimenting with the Capacity scheduler (0.19.0) in a multi-cluster
environment.
I noticed that unlike the mappers, the reducers are not pre-empted?

I have two queues (high and low) that are each running big jobs (70+ maps
each).  The scheduler splits the mappers as per the queue
guaranteed-capacity (5/8ths for the high and the rest for the low). However,
the reduce jobs are not interleaved -- the reduce job in the high queue is
blocked waiting for the reduce job in the low queue to complete.

Is this a bug or by design?

*Low queue:*
Guaranteed Capacity (%) : 37.5
Guaranteed Capacity Maps : 3
Guaranteed Capacity Reduces : *3*
User Limit : 100
Reclaim Time limit : 300
Number of Running Maps : 3
Number of Running Reduces : *7*
Number of Waiting Maps : 131
Number of Waiting Reduces : 0
Priority Supported : NO

*High queue:*
Guaranteed Capacity (%) : 62.5
Guaranteed Capacity Maps : 5
Guaranteed Capacity Reduces : 5
User Limit : 100
Reclaim Time limit : 300
Number of Running Maps : 4
Number of Running Reduces : *0*
Number of Waiting Maps : 68
Number of Waiting Reduces : *7*
Priority Supported : NO

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message