hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurabh Bajaj <Saurabh.Ba...@mu-sigma.com>
Subject All reducers are not being utilized
Date Thu, 02 Aug 2012 10:25:13 GMT
Hi everyone,

I was running a MR job in java and this scenario happened:

Case 1:

Number of distinct output keys from mapper = 3

Expected # of reducers = 3

Defined set # of reducers to be called = 2


Expected outcome:

# of reducers spawned = 2

# of keys processed under first reducer = 1

# of keys processed under second reducer = 2


Observed outcome:

# of keys processed under first reducer = 3

# of keys processed under second reducer = 0




Case 2:

Number of distinct output keys from mapper = 3

Expected # of reducers = 3

Defined set # of reducers to be called = 3


Expected outcome:

# of reducers spawned = 3

# of keys processed under first reducer = 1

# of keys processed under second reducer = 1
# of keys processed under third reducer = 1


Observed outcome:

# of reducers spawned = 3

# of keys processed under first reducer = 2

# of keys processed under second reducer = 0
# of keys processed under third reducer = 1



Any idea why all the reducers are not utilized?

Saurabh Bajaj | Senior Business Analyst | +91 9986588089 | www.mu-sigma.com<http://www.mu-sigma.com/>
|


________________________________
This email message may contain proprietary, private and confidential information. The information
transmitted is intended only for the person(s) or entities to which it is addressed. Any review,
retransmission, dissemination or other use of, or taking of any action in reliance upon, this
information by persons or entities other than the intended recipient is prohibited and may
be illegal. If you received this in error, please contact the sender and delete the message
from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic communications are free
from viruses. However, given Internet accessibility, the Company cannot accept liability for
any virus introduced by this e-mail or any attachment and you are advised to use up-to-date
virus checking software.

Mime
View raw message