hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bejoy KS" <bejoy.had...@gmail.com>
Subject Re: number of reducers
Date Tue, 20 Nov 2012 20:09:30 GMT
Hi Sasha

By default the number or reducers are set to be 1. If you want more you need to specify it

hadoop jar myJar.jar myClass -D mapred.reduce.tasks=20 ...

Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: jamal sasha <jamalshasha@gmail.com>
Date: Tue, 20 Nov 2012 14:38:54 
To: <user@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: number of reducers


  I wrote a simple map reduce job in hadoop streaming.

I am wondering if I am doing something wrong ..

While number of mappers are projected to be around 1700.. reducers.. just 1?

It’s couple of TB’s worth of data.

What can I do to address this.

Basically mapper looks like this

For line in sys.stdin:

    Print line


For line in sys.stdin:

    New_line = process_line(line)

    Print new_line


View raw message