hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
Date Wed, 31 Aug 2011 14:22:09 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094562#comment-13094562

Robert Joseph Evans commented on MAPREDUCE-2876:

I am fine with the polling like I said before.  The question I was trying to answer was what
is the ramifications of not having a polling interval, but just having an expiry interval,
and setting the polling interval off of the expiry interval.  It means the real expiry interval
is any time between expiryInterval and expiryInterval + pollingInterval.  If that is never
a problem then polling is fine. 

I can see your reasoning behind the change.  When there are N objects being tracked then polling
would be

 O((pollingFrequency * N) + (modificationFrequency * N))

But Priority Queue would be

O(modificationFrequency * N * log(N) + expiryFrequency)

So if the modificationFrequency is very large then polling definitely wins in complexity,
but I am curious did you run any benchmarks to see what kind of an extra load would be placed
on the system with a priority queue (Just curious what it might be)?

> ContainerAllocationExpirer appears to use the incorrect configs
> ---------------------------------------------------------------
>                 Key: MAPREDUCE-2876
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>             Fix For: 0.23.0
> ContainerAllocationExpirer sets the expiration interval to be RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL
but uses AMLIVELINESS_MONITORING_INTERVAL as the interval.  This is very different from what
AMLivelinessMonitor does.
> There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL for the
monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the expiry.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message