hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartashov, Andy" <Andy.Kartas...@mpac.ca>
Subject RE: number of reducers
Date Tue, 20 Nov 2012 21:50:44 GMT
I specify mine inside mapred-site.xml

<property>
    <name>mapred.reduce.tasks</name>
    <value>20</value>
  </property>

Rgds,
AK47
From: Bejoy KS [mailto:bejoy.hadoop@gmail.com]
Sent: Tuesday, November 20, 2012 3:10 PM
To: user@hadoop.apache.org
Subject: Re: number of reducers

Hi Sasha

By default the number or reducers are set to be 1. If you want more you need to specify it
as

hadoop jar myJar.jar myClass -D mapred.reduce.tasks=20 ...
Regards
Bejoy KS

Sent from handheld, please excuse typos.
________________________________
From: jamal sasha <jamalshasha@gmail.com>
Date: Tue, 20 Nov 2012 14:38:54 -0500
To: <user@hadoop.apache.org>
ReplyTo: user@hadoop.apache.org
Subject: number of reducers



Hi,

  I wrote a simple map reduce job in hadoop streaming.



I am wondering if I am doing something wrong ..

While number of mappers are projected to be around 1700.. reducers.. just 1?

It's couple of TB's worth of data.

What can I do to address this.

Basically mapper looks like this



For line in sys.stdin:

    Print line



Reducer

For line in sys.stdin:

    New_line = process_line(line)

    Print new_line





Thanks

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe
qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite.
Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement
l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

Mime
View raw message