hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alx...@aim.com
Subject Re: number of reducers
Date Tue, 20 Nov 2012 22:00:57 GMT

 What is the relationship between number of reducers and cpu cores in your setup? I read somewhere
that it must be .5 of number of cpu cores.




-----Original Message-----
From: Kartashov, Andy <Andy.Kartashov@mpac.ca>
To: user <user@hadoop.apache.org>; bejoy.hadoop <bejoy.hadoop@gmail.com>
Sent: Tue, Nov 20, 2012 1:51 pm
Subject: RE: number of reducers

I specify mine inside mapred-site.xml

From: Bejoy KS [mailto:bejoy.hadoop@gmail.com]
Sent: Tuesday, November 20, 2012 3:10 PM
To: user@hadoop.apache.org
Subject: Re: number of reducers

Hi Sasha

By default the number or reducers are set to be 1. If you want more you need to specify it

hadoop jar myJar.jar myClass -D mapred.reduce.tasks=20 ...

Bejoy KS

Sent from handheld, please excuse typos.

From: jamal sasha <jamalshasha@gmail.com> 

Date: Tue, 20 Nov 2012 14:38:54 -0500

To: <user@hadoop.apache.org>

ReplyTo: user@hadoop.apache.org 

Subject: number of reducers



  I wrote a simple map reduce job in hadoop streaming.


I am wondering if I am doing something wrong ..

While number of mappers are projected to be around 1700.. reducers.. just 1?

It’s couple of TB’s worth of data.

What can I do to address this.

Basically mapper looks like this


For line in sys.stdin:

    Print line



For line in sys.stdin:

    New_line = process_line(line)

    Print new_line




NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe
qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite.
Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

View raw message