Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E575E10724 for ; Thu, 4 Dec 2014 05:31:13 +0000 (UTC) Received: (qmail 31892 invoked by uid 500); 4 Dec 2014 05:31:13 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 31832 invoked by uid 500); 4 Dec 2014 05:31:13 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 31820 invoked by uid 99); 4 Dec 2014 05:31:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Dec 2014 05:31:13 +0000 Date: Thu, 4 Dec 2014 05:31:13 +0000 (UTC) From: "Naganarasimha G R (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233925#comment-14233925 ] Naganarasimha G R commented on YARN-2874: ----------------------------------------- Hi [~kasha] & [~ozawa] Thanks for reviewing and commiting the patch . > Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps > --------------------------------------------------------------------------------- > > Key: YARN-2874 > URL: https://issues.apache.org/jira/browse/YARN-2874 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Affects Versions: 2.6.0, 2.5.1 > Reporter: Naganarasimha G R > Assignee: Naganarasimha G R > Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-2874.20141118-1.patch, YARN-2874.20141118-2.patch > > > When token renewal fails and the application finishes this dead lock can occur > Jstack dump : > {quote} > Found one Java-level deadlock: > ============================= > "DelegationTokenRenewer #181865": > waiting to lock monitor 0x0000000000900918 (object 0x00000000c18a9998, a java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > "DelayedTokenCanceller": > waiting to lock monitor 0x0000000004141718 (object 0x00000000c7eae720, a org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask), > which is held by "Timer-4" > "Timer-4": > waiting to lock monitor 0x0000000000900918 (object 0x00000000c18a9998, a java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > > Java stack information for the threads listed above: > =================================================== > "DelegationTokenRenewer #181865": > at java.util.Collections$SynchronizedCollection.add(Collections.java:1636) > - waiting to lock <0x00000000c18a9998> (a java.util.Collections$SynchronizedSet) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DelayedTokenCanceller": > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443) > - waiting to lock <0x00000000c7eae720> (a org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558) > - locked <0x00000000c18a9998> (a java.util.Collections$SynchronizedSet) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599) > at java.lang.Thread.run(Thread.java:745) > "Timer-4": > at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - waiting to lock <0x00000000c18a9998> (a java.util.Collections$SynchronizedSet) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70) > at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437) > - locked <0x00000000c7eae720> (a org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Found 1 deadlock. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)