hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eduard Skaley <e.v.ska...@gmail.com>
Subject Re: Controlling on which node a reducer will be executed
Date Wed, 29 Aug 2012 11:31:41 GMT
Thank you for your reply Harsh J.
I really need this feature, because this will boost up execution of my use case 
a lot.
Could you give me a hint where to look in the sources to get a good starting 
point for implementation?
Which classes are involved?

> Hi Eduard,
>
> This isn't impossible, just unavailable at the moment. See
> https://issues.apache.org/jira/browse/MAPREDUCE-199 if you want to
> take a shot at implementing this. HBase would love to have this, I
> think.
>
> On Mon, Aug 27, 2012 at 10:41 PM, Eduard Skaley <e.v.skaley@gmail.com> wrote:
>> Hi,
>>
>> i have a question concerning the execution of reducers.
>> To use effectively the data locality of blocks in my use case i want to
>> control on which node a reducer will be executed.
>>
>> In my scenario i have a chain of map-reduce jobs where each job will be
>> executed by exactly N reducers.
>> I want to achieve that for each job reducer 1 is executed on node 1,
>> reducer 2 on node 2 and so on.
>>
>> Is it possible with a customization of the BlockPlacementPolicy in MRv2 ?
>> Or is there some other way, maybe some indirect ? Or is it impossible ?
>>
>> Thanks for your help.
>> Eduard
>
>


Mime
View raw message