Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 21444 invoked from network); 2 Feb 2007 17:59:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 2 Feb 2007 17:59:30 -0000 Received: (qmail 12636 invoked by uid 500); 2 Feb 2007 17:59:34 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 12609 invoked by uid 500); 2 Feb 2007 17:59:34 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 12342 invoked by uid 99); 2 Feb 2007 17:59:34 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Feb 2007 09:59:33 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Feb 2007 09:59:26 -0800 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 0193F71429F for ; Fri, 2 Feb 2007 09:59:06 -0800 (PST) Message-ID: <25999927.1170439146003.JavaMail.jira@brutus> Date: Fri, 2 Feb 2007 09:59:06 -0800 (PST) From: "Owen O'Malley (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Created: (HADOOP-969) deadlock in job tracker RetireJobs MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org deadlock in job tracker RetireJobs ---------------------------------- Key: HADOOP-969 URL: https://issues.apache.org/jira/browse/HADOOP-969 Project: Hadoop Issue Type: Bug Components: mapred Reporter: Owen O'Malley Assigned To: Owen O'Malley Priority: Critical Fix For: 0.11.0 The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like: "IPC Server handler 5 on 50020": at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108) - waiting to lock <0x74487a80> (a java.util.Vector) - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker) at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992) - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538) "retireJobs": at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782) - waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker) at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42) at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312) - locked <0x74487bb0> (a java.util.ArrayList) - locked <0x74487a80> (a java.util.Vector) - locked <0x74487a58> (a java.util.TreeMap) at java.lang.Thread.run(Thread.java:595) Found 1 deadlock. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.