hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Baldeschwieler <jeri...@me.com>
Subject Re: [jira] [Resolved] (MAPREDUCE-2694) AM releases too many containers due to the protocol
Date Mon, 18 Jul 2011 03:48:03 GMT
this seems very odd.  Shouldn't we be tracking how many are assigned, never letting the assigned
number rise over 7?

On Jul 16, 2011, at 3:13 PM, Arun C Murthy (JIRA) wrote:

> 
>     [ https://issues.apache.org/jira/browse/MAPREDUCE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
> 
> Arun C Murthy resolved MAPREDUCE-2694.
> --------------------------------------
> 
>    Resolution: Not A Problem
> 
> I thought more about this... I've spent time on this in the beginning and I don't think
a delta protocol is appropriate - particularly since the RM and AM aren't 'in sync' ever.
> 
> Yes, this could lead to some waste, but the system is eventually consistent.
> 
>> AM releases too many containers due to the protocol
>> ---------------------------------------------------
>> 
>>                Key: MAPREDUCE-2694
>>                URL: https://issues.apache.org/jira/browse/MAPREDUCE-2694
>>            Project: Hadoop Map/Reduce
>>         Issue Type: Bug
>>         Components: mrv2
>>           Reporter: Arun C Murthy
>>           Assignee: Arun C Murthy
>> 
>> - AM sends request asking 4 containers on host H1.
>> - Asynchronously, host H1 reaches RM and gets assigned 4 containers. RM at this point,
sets the value against H1 to
>> zero in its aggregate request-table for all apps.
>> - In the mean-while AM gets to need 3 more containers, so a total of 7 including
the 4 from previous request.
>> - Today, AM sends the absolute number of 7 against H1 to RM as part of its request
table.
>> - RM seems to be overriding its earlier value of zero against H1 to 7 against H1.
And thus allocating 7 more
>> containers.
>> - AM already gets 4 in this scheduling iteration, but gets 7 more, a total of 11
instead of the required 7.
> 
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
> 
> 


Mime
View raw message