hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Ideas for dynamic change reducer task number ?
Date Tue, 24 Nov 2009 00:36:07 GMT
You're right, I will try that.

Thank you


Jeff Zhang


On Mon, Nov 23, 2009 at 9:19 AM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Nov 22, 2009, at 4:48 PM, Jeff Zhang wrote:
>
>  My concern is that it is just like hard code to use conf.setNumReduceTasks
>> on the configuration. It is not flexible, so my idea is that adding an
>> interface to change the reducer number dynamically according the different
>> size of input data set.
>>
>
> You misunderstand. I meant doing something like:
>
> public class MyInputFormat ....
>
>  public InputSplit[] getSplits(JobConf conf) {
>     InputSplit[] result = ...;
>     // compute total size of input
>     conf.setNumReduceTasks(max(6, size / 10G));
>  }
> }
>
> I haven't checked the code to make sure it will work, but I believe it
> will.
>
> -- Owen
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message