Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E6B2917A05 for ; Thu, 25 Sep 2014 21:40:34 +0000 (UTC) Received: (qmail 61377 invoked by uid 500); 25 Sep 2014 21:40:34 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 61337 invoked by uid 500); 25 Sep 2014 21:40:34 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 61326 invoked by uid 99); 25 Sep 2014 21:40:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Sep 2014 21:40:34 +0000 Date: Thu, 25 Sep 2014 21:40:34 +0000 (UTC) From: "Karthik Kambatla (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2608) FairScheduler may hung due to two potential deadlocks MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148354#comment-14148354 ] Karthik Kambatla commented on YARN-2608: ---------------------------------------- +1, pending Jenkins. > FairScheduler may hung due to two potential deadlocks > ----------------------------------------------------- > > Key: YARN-2608 > URL: https://issues.apache.org/jira/browse/YARN-2608 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Wei Yan > Assignee: Wei Yan > Attachments: YARN-2608-1.patch, YARN-2608-2.patch > > > Two potential deadlocks exist inside the FairScheduler. > 1. AllocationFileLoaderService would reload the queue configuration, which calls FairScheduler.AllocationReloadListener.onReload() function. And require *FairScheduler's lock*; > {code} > public void onReload(AllocationConfiguration queueInfo) { > synchronized (FairScheduler.this) { > .... > } > } > {code} > after that, it would require the *QueueManager's queues lock*. > {code} > private FSQueue getQueue(String name, boolean create, FSQueueType queueType) { > name = ensureRootPrefix(name); > synchronized (queues) { > .... > } > } > {code} > Another thread FairScheduler.assignToQueue may also need to create a new queue when a new job submitted. This thread would hold the *QueueManager's queues lock* firstly, and then would like to hold the *FairScheduler's lock* as it needs to call FairScheduler.getClock() function when creating a new FSLeafQueue. Deadlock may happen here. > 2. The AllocationFileLoaderService holds *AllocationFileLoaderService's lock* first, and then waits for *FairScheduler's lock*. Another thread (like AdminService.refreshQueues) may call FairScheduler's reinitialize function, which holds *FairScheduler's lock* first, and then waits for *AllocationFileLoaderService's lock*. Deadlock may happen here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)