hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-270) RM scheduler event handler thread gets behind
Date Thu, 13 Dec 2012 19:00:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531331#comment-13531331
] 

Thomas Graves commented on YARN-270:
------------------------------------

I was able to reproduce this on a smaller cluster by simulating 2800 nodes - had 720 node
manager and made the heartbeat internal 250 ms (instead of 1s).  At about 400 applications
the scheduler queue starts to grow.  I am still in the process of investigating what exactly
is taking the time.

Note that for now we believe we have worked around it by increasing the nodemanager heartbeat
interval from 1 second to 3 seconds.
                
> RM scheduler event handler thread gets behind
> ---------------------------------------------
>
>                 Key: YARN-270
>                 URL: https://issues.apache.org/jira/browse/YARN-270
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 0.23.5
>            Reporter: Thomas Graves
>
> We had a couple of incidents on a 2800 node cluster where the RM scheduler event handler
thread got behind processing events and basically become unusable.  It was still processing
apps, but taking a long time (1 hr 45 minutes) to accept new apps.   this actually happened
twice within 5 days.
> We are using the capacity scheduler and at the time had between 400 and 500 applications
running.  There were another 250 apps that were in the SUBMITTED state in the RM but the scheduler
hadn't processed those to put in pending state yet.  We had about 15 queues none of them hierarchical.
 We also had plenty of space lefts on the cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message