hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4089) Race condition when calling AbstractYarnScheduler.completedContainer.
Date Thu, 27 Aug 2015 13:12:47 GMT

    [ https://issues.apache.org/jira/browse/YARN-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716629#comment-14716629
] 

Hadoop QA commented on YARN-4089:
---------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  16m 45s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear to include any
new or modified tests.  Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. |
| {color:green}+1{color} | javac |   7m 59s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |   9m 57s | There were no new javadoc warning messages.
|
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   0m 51s | The applied patch generated  1 new checkstyle
issues (total was 70, now 71). |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that end in whitespace.
|
| {color:green}+1{color} | install |   1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with eclipse:eclipse.
|
| {color:green}+1{color} | findbugs |   1m 28s | The patch does not introduce any new Findbugs
(version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests |  53m 30s | Tests passed in hadoop-yarn-server-resourcemanager.
|
| | |  92m 57s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12752718/YARN-4089.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0bf2854 |
| checkstyle |  https://builds.apache.org/job/PreCommit-YARN-Build/8927/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8927/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8927/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8927/console |


This message was automatically generated.

> Race condition when calling AbstractYarnScheduler.completedContainer.
> ---------------------------------------------------------------------
>
>                 Key: YARN-4089
>                 URL: https://issues.apache.org/jira/browse/YARN-4089
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.6.0, 2.7.0, 2.5.2, 2.7.1
>            Reporter: Shiwei Guo
>              Labels: patch
>         Attachments: YARN-4089.001.patch
>
>
> There is a  race condition of calling AbstractYarnScheduler.completedContainer, which
will cause the usedResource counter of application not accurate. At worst situation, the scheduler
will not allocate any resource to any application in some queue( when the usedResource became
negative) even there is indeed lots of free resource to be allocated.
> It also cause the Scheduler UI and metrics report negative resource usage value.In our
cluster, it has the ability to run 13000+ container, but the WEB UI says that:
> - Containers Running: -26546
> - Memory Used: -82.38 TB
> - VCores Used: -26451
> This is how it happens in FairSchedular:
> completedContainer method will call application.containerCompleted, which will subtraction
the resources used by this container from the usedResource counter of the application. So,
if the completedContainer are called twice with the same container, the counter is subtracted
too much values. So is the updateRootQueueMetrics call, so we can see negative allocatedMemory
on rootQueue.
> The solution is to check whether the container being supplied is still live inside the
completedContainer (as shown in the patch). There is some check before calling completedContainer,
but that's not enough.
> For a more deeply discussion, the completedContainer may be called from two place:
> 1. Trigered by RMContainerEventType.FINISHED event:
> {code:title=FairScheduler.nodeUpdate}
> // Process completed containers
>     for (ContainerStatus completedContainer : completedContainers) {
>       ContainerId containerId = completedContainer.getContainerId();
>       LOG.debug("Container FINISHED: " + containerId);
>       completedContainer(getRMContainer(containerId),
>           completedContainer, RMContainerEventType.FINISHED);
>     }
> {code}
> 2. Trigered by RMContainerEventType.RELEASED
> {code:title=AbstractYarnScheduler.releaseContainers}
> completedContainer(rmContainer,
>         SchedulerUtils.createAbnormalContainerStatus(containerId,
>           SchedulerUtils.RELEASED_CONTAINER), RMContainerEventType.RELEASED);
> {code}
> RMContainerEventType.RELEASED is not triggered by MapReduce ApplicationMaster, so we
won't see this problem on MR jobs. But TEZ will triggered it when it do not need this this
container, while the NodeManger will also report a container complete message to RM ,which
in turn trigger the RMContainerEventType.FINISHED event. If RMContainerEventType.FINISHED
event comes to RM early than TEZ AM, the problem happens.
> This behavior can be more easily seen if the cluster had setup a TimelineServer for TEZ,
which make it more likely TEZ AM will send RMContainerEventType.RELEASED event later than
NM send RMContainerEventType.FINISHED.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message