Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 2 Jun 2016 00:44:59 +0000 (UTC)
From: "Junping Du (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12974842.1464796250000.9574.1464828299229@Atlassian.JIRA>
In-Reply-To: <JIRA.12974842.1464796250000@Atlassian.JIRA>
References: <JIRA.12974842.1464796250000@Atlassian.JIRA> <JIRA.12974842.1464796250263@arcas>
Subject: [jira] [Commented] (YARN-5190) Race condition in registering
 container metrics cause uncaught exception in ContainerMonitorImpl
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 02 Jun 2016 00:45:01 -0000


    [ https://issues.apache.org/jira/browse/YARN-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311496#comment-15311496 ] 

Junping Du commented on YARN-5190:
----------------------------------

Discussed offline with [~jianhe], we think a couple of things need to get fixed here :
1. Fix the asymmetric behaviors in register()/unregisterSource() at MetricsSystemImpl that source name is still left in {{sourceNames.map}} in DefaultMetricsSystem after unregisterSource().

2. ContainerMetrics.finished() could get called twice - one for container life cycle (involved in YARN-4906) and the other in container monitoring life cycle. Ideally, it is better to make sure ContainerMetrics.finished() for the same container only get called one time one place. However, in practice, the container event life cycle and container monitor event life cycle are independent and cannot replace each other. Alternatively, we will make sure scheduleTimerTaskForUnregistration() only get called one time or it will be more threads of unregistration than needed.

3. In case one ContainerMetrics already get finished before (triggered by Container life cycle event), current logic in {{ContainerMonitorImpl.updateContainerMetrics(ContainersMonitorEvent)}} will still register metrics into DefaultMetricsSystem first (via ContainerMetrics.forContainer(...)) and unregister it from DefaultMetricsSystem soon after. This is completely unnecessary.

Will deliver a fix for three issues raised above.

> Race condition in registering container metrics cause uncaught exception in ContainerMonitorImpl
> ------------------------------------------------------------------------------------------------
>
>                 Key: YARN-5190
>                 URL: https://issues.apache.org/jira/browse/YARN-5190
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Junping Du
>            Assignee: Junping Du
>            Priority: Blocker
>
> The exception stack is as following:
> {noformat}
> 310735 2016-05-22 01:50:04,554 [Container Monitor] ERROR org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Container Monitor,5,main] threw an Exception.
> 310736 org.apache.hadoop.metrics2.MetricsException: Metrics source ContainerResource_container_1463840817638_14484_01_000010 already exists!
> 310737         at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:135)
> 310738         at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:112)
> 310739         at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> 310740         at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.forContainer(ContainerMetrics.java:212)
> 310741         at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.forContainer(ContainerMetrics.java:198)
> 310742         at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:385)
> {noformat}
> After YARN-4906, we have multiple places to get ContainerMetrics for a particular container that could cause race condition in registering the same container metrics to DefaultMetricsSystem by different threads. Lacking of proper handling of MetricsException which could get thrown, the exception will could bring down daemon of ContainerMonitorImpl or even whole NM.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org