hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shiwei Guo (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-4089) Race condition when calling AbstractYarnScheduler.completedContainer.
Date Thu, 27 Aug 2015 11:30:45 GMT
Shiwei Guo created YARN-4089:
--------------------------------

             Summary: Race condition when calling AbstractYarnScheduler.completedContainer.
                 Key: YARN-4089
                 URL: https://issues.apache.org/jira/browse/YARN-4089
             Project: Hadoop YARN
          Issue Type: Bug
          Components: resourcemanager
    Affects Versions: 2.7.1, 2.5.2, 2.7.0, 2.6.0
            Reporter: Shiwei Guo


There is a  race condition of calling AbstractYarnScheduler.completedContainer, which will
cause the usedResource counter of application not accurate. At worst situation, the scheduler
will not allocate any resource to any application in some queue( when the usedResource became
negative) even there is indeed lots of free resource to be allocated.

It also cause the Scheduler UI and metrics report negative resource usage value.In our cluster,
it has the ability to run 13000+ container, but the WEB UI says that:

- Containers Running: -26546
- Memory Used: -82.38 TB
- VCores Used: -26451

This is how it happens in FairSchedular:

completedContainer method will call application.containerCompleted, which will subtraction
the resources used by this container from the usedResource counter of the application. So,
if the completedContainer are called twice with the same container, the counter is subtracted
too much values. So is the updateRootQueueMetrics call, so we can see negative allocatedMemory
on rootQueue.

The solution is to check whether the container being supplied is still live inside the completedContainer
(as shown in the patch). There is some check before calling completedContainer, but that's
not enough.

For a more deeply discussion, the completedContainer may be called from two place:

1. Trigered by RMContainerEventType.FINISHED event:

{code:title=FairScheduler.nodeUpdate}
// Process completed containers
    for (ContainerStatus completedContainer : completedContainers) {
      ContainerId containerId = completedContainer.getContainerId();
      LOG.debug("Container FINISHED: " + containerId);
      completedContainer(getRMContainer(containerId),
          completedContainer, RMContainerEventType.FINISHED);
    }
{code}

2. Trigered by RMContainerEventType.RELEASED

{code:title=AbstractYarnScheduler.releaseContainers}
completedContainer(rmContainer,
        SchedulerUtils.createAbnormalContainerStatus(containerId,
          SchedulerUtils.RELEASED_CONTAINER), RMContainerEventType.RELEASED);
{code}

RMContainerEventType.RELEASED is not triggered by MapReduce ApplicationMaster, so we won't
see this problem on MR jobs. But TEZ will triggered it when it do not need this this container,
while the NodeManger will also report a container complete message to RM ,which in turn trigger
the RMContainerEventType.FINISHED event. If RMContainerEventType.FINISHED event comes to RM
early than TEZ AM, the problem happens.

This behavior can be more easily seen if the cluster had setup a TimelineServer for TEZ, which
make it more likely TEZ AM will send RMContainerEventType.RELEASED event later than NM send
RMContainerEventType.FINISHED.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message