hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Wrigley <...@cloudera.com>
Subject Re: secondary sort - number of reducers
Date Thu, 29 Aug 2013 23:38:27 GMT
If you don't specify the number of Reducers, Hadoop will use the default -- which, unless you've
changed it, is 1.

Regards

Ian.

On Aug 29, 2013, at 4:23 PM, Adeel Qureshi <adeelmahmood@gmail.com> wrote:

> I have implemented secondary sort in my MR job and for some reason if i dont specify
the number of reducers it uses 1 which doesnt seems right because im working with 800M+ records
and one reducer slows things down significantly. Is this some kind of limitation with the
secondary sort that it has to use a single reducer .. that kind of would defeat the purpose
of having a scalable solution such as secondary sort. I would appreciate any help.
> 
> Thanks
> Adeel


---
Ian Wrigley
Sr. Curriculum Manager
Cloudera, Inc
Cell: (323) 819 4075


Mime
View raw message