hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: number of running reducers exceed max number of mappers on cluster
Date Thu, 01 Dec 2011 02:58:44 GMT
For how to control Pig's parallelism: http://pig.apache.org/docs/r0.9.1/perf.html#parallel

For your 'running' task count bug: https://issues.apache.org/jira/browse/MAPREDUCE-486 (Which
would mean that only 5 may be 'running', while the rest may have already completed and were
awaiting finalization of outputs. Not to be worried about - its more of a UI bug.)

On 01-Dec-2011, at 6:36 AM, Keren Ouaknine wrote:

> Pig script allocates 15 nodes to run a pig script (how to configure that?) but what happens
on a 10 nodes cluster only?
> UI shows that 15 reducers are running, and at the same time max number of reducers is
10 (configured one reducer per machine).
> 
> 
> 
> Thanks,
> Keren
> 
> Kind	% Complete	Num Tasks	Pending	Running	Complete	Killed	Failed/Killed
> Task Attempts
> map	100.00%
> 
> 1	0	0	1	0	0 / 0
> reduce	5.92%
> 
> 
> 90	70	15	5	0	0 / 0
> 
> Running Map Tasks	Running Reduce Tasks	Total Submissions	 Nodes	Occupied Map Slots	Occupied
Reduce Slots	Reserved Map Slots	Reserved Reduce Slots	Map Task Capacity	Reduce Task Capacity
Avg. Tasks/Node	Blacklisted Nodes	Graylisted Nodes	Excluded Nodes
> 0	0	49	10	0	0	0	0	10	10	2.00	0	0	0
> 
> 
> 
> 
> -- 
> Keren Ouaknine
> Cell: +972 54 2565404
> Web: www.kereno.com
> 
> 


Mime
View raw message