hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4408) NodeManager still reports negative running containers
Date Wed, 02 Dec 2015 18:42:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15036345#comment-15036345
] 

Robert Kanter commented on YARN-4408:
-------------------------------------

I haven't been able to reproduce this issue, and I agree that it's not a common occurrence;
but we have seen the number of running containers go negative internally on two different
clusters and also on a customer's cluster.  So I started through the code and state machine
for how we could decrement the gauge without first incrementing it.  As far as I can tell,
this is the only way where this can happen because we don't check {{container.wasLaunched}}
like in the other two places where we decrement the gauge.  

> NodeManager still reports negative running containers
> -----------------------------------------------------
>
>                 Key: YARN-4408
>                 URL: https://issues.apache.org/jira/browse/YARN-4408
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.4.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: YARN-4408.001.patch
>
>
> YARN-1697 fixed a problem where the NodeManager metrics could report a negative number
of running containers.  However, it missed a rare case where this can still happen.
> YARN-1697 added a flag to indicate if the container was actually launched ({{LOCALIZED}}
to {{RUNNING}}) or not ({{LOCALIZED}} to {{KILLING}}), which is then checked when transitioning
from {{CONTAINER_CLEANEDUP_AFTER_KILL}} to {{DONE}} and {{EXITED_WITH_FAILURE}} to {{DONE}}
to only decrement the gauge if we actually ran the container and incremented the gauge . 
However, this flag is not checked while transitioning from {{EXITED_WITH_SUCCESS}} to {{DONE}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message