hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: max 1 mapper per node
Date Fri, 27 Apr 2012 15:30:44 GMT

You would want to modify the application master for this, and it is likely to be a bit of
a hack because the RM scheduler itself is not really designed for this.  You can have the
AM return containers to the RM that it does not want (which it does already in some cases),
because they are on the same node as an already running map task.  This could slow down your
application a lot, as it makes it more difficult for the RM to find free containers that the
AM wants.  You could also modify the AM so that it will not request any more then just one
container per node, which would help with this, but not solve it completely.

--Bobby Evans

On 4/27/12 12:38 AM, "Radim Kolar" <hsn@filez.com> wrote:

Do you want it so that the node can run only 1 mapper at a time but the node can also run
multiple reducers at the same time?
I currently want it per-application. Node can run more mappers/reducers from other jobs at
same time.

 Do you want it so that this particular application will only use that node for a single mapper,
but other applications may also run mappers on the node?
yes. After this will be coded, i will rerun tests if it is sufficient for other types of tasks

 For start, i would need per job settings - max mappers/reducers per node, so modifying AM
would be enough.

View raw message