hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (YARN-4047) ClientRMService getApplications has high scheduler lock contention
Date Tue, 11 Aug 2015 19:53:46 GMT

     [ https://issues.apache.org/jira/browse/YARN-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jason Lowe reassigned YARN-4047:

    Assignee: Jason Lowe

In OOZIE-1729 Oozie started calling getApplications to look for applications with specific
tags.  This significantly increases the utilization of this method on a cluster that makes
heavy use of Oozie.

One quick fix for the Oozie use-case may be to swap the filter order.  Rather than doing the
expensive checkAccess call first, we can do all the other filtering first and finally verify
the user has access before adding the app to the response.  In the Oozie scenario most apps
will be filtered by the tag check before we ever get to the checkAccess call.

> ClientRMService getApplications has high scheduler lock contention
> ------------------------------------------------------------------
>                 Key: YARN-4047
>                 URL: https://issues.apache.org/jira/browse/YARN-4047
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
> The getApplications call can be particuarly expensive because the code can call checkAccess
on every application being tracked by the RM.  checkAccess will often call scheduler.checkAccess
which will grab the big scheduler lock.  This can cause a lot of contention with the scheduler
thread which is busy trying to process node heartbeats, app allocation requests, etc.

This message was sent by Atlassian JIRA

View raw message