hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-969) deadlock in job tracker RetireJobs
Date Fri, 02 Feb 2007 18:40:05 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley updated HADOOP-969:
---------------------------------

    Attachment: jt-dead.patch

This code has gotten pretty hairy, but this patch goes through the list of jobs to find ones
that need to be retired. It then locks everything including the JobTracker first and does
the mutations to all of the maps. We really need to refactor the datastructures at some point
soon...

> deadlock in job tracker RetireJobs
> ----------------------------------
>
>                 Key: HADOOP-969
>                 URL: https://issues.apache.org/jira/browse/HADOOP-969
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>            Priority: Critical
>             Fix For: 0.11.0
>
>         Attachments: jt-dead.patch
>
>
> The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call
stacks look like:
> "IPC Server handler 5 on 50020":
>        at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)
>        - waiting to lock <0x74487a80> (a java.util.Vector)
>        - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
>        at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
>        - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
>        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
> "retireJobs":
>        at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
>        - waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
>        at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
>        at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
>        - locked <0x74487bb0> (a java.util.ArrayList)
>        - locked <0x74487a80> (a java.util.Vector)
>        - locked <0x74487a58> (a java.util.TreeMap)
>        at java.lang.Thread.run(Thread.java:595)
> Found 1 deadlock.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message