hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1436) Deadlock in preemption code in fair scheduler
Date Thu, 04 Feb 2010 07:14:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829508#action_12829508

Matei Zaharia commented on MAPREDUCE-1436:

Hi Todd,

It looks like that particular problem won't happen with a real JobTracker because the JobTracker
only calls listener.jobAdded/jobRemoved when it is already holding a lock on itself (e.g.
in JobTracker.addJob). However, it might not hurt to acquire the lock in FairScheduler, in
case this JobTracker behavior changes. Do you think it's better to do that, or to "fix" the
fake TaskTrackerManager?

In older versions of the fair scheduler, I *always* locked the JT before locking the scheduler.
Some of the Yahoo guys removed this because they said it led to scalability issues, though
maybe that isn't a problem anymore.

> Deadlock in preemption code in fair scheduler
> ---------------------------------------------
>                 Key: MAPREDUCE-1436
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1436
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Matei Zaharia
>            Assignee: Matei Zaharia
>            Priority: Blocker
>         Attachments: deadlock.png, mapreduce-1436.patch
> In testing the fair scheduler with preemption, I found a deadlock between updatePreemptionVariables
and some code in the JobTracker. This was found while testing a backport of the fair scheduler
to Hadoop 0.20, but it looks like it could also happen in trunk and 0.21. Details are in a
comment below.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message