hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Ideas for dynamic change reducer task number ?
Date Fri, 27 Nov 2009 07:31:46 GMT
Owen,

It works, thank you for your help.


Jeff Zhang



On Tue, Nov 24, 2009 at 8:36 AM, Jeff Zhang <zjffdu@gmail.com> wrote:

>
> You're right, I will try that.
>
> Thank you
>
>
> Jeff Zhang
>
>
>
> On Mon, Nov 23, 2009 at 9:19 AM, Owen O'Malley <omalley@apache.org> wrote:
>
>>
>> On Nov 22, 2009, at 4:48 PM, Jeff Zhang wrote:
>>
>>  My concern is that it is just like hard code to use
>>> conf.setNumReduceTasks
>>> on the configuration. It is not flexible, so my idea is that adding an
>>> interface to change the reducer number dynamically according the
>>> different
>>> size of input data set.
>>>
>>
>> You misunderstand. I meant doing something like:
>>
>> public class MyInputFormat ....
>>
>>  public InputSplit[] getSplits(JobConf conf) {
>>     InputSplit[] result = ...;
>>     // compute total size of input
>>     conf.setNumReduceTasks(max(6, size / 10G));
>>  }
>> }
>>
>> I haven't checked the code to make sure it will work, but I believe it
>> will.
>>
>> -- Owen
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message