hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3519) Deadlock in LocalDirsHandlerService and ShuffleHandler
Date Fri, 09 Dec 2011 15:02:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166218#comment-13166218

Robert Joseph Evans commented on MAPREDUCE-3519:

I like the patch.  I also traced down from where the synchronized call is happening looking
for a potential race condition between reads and writes.  The only place I found that uses
conf from LocalDirsHandlerService.getLocalPathForWrite is LocalDirAllocator.confChanged. 
Inside that code we access conf three times, once to get the local file system, so I don't
see that as being an issue.  But we call it twice to get the value of the contextConfigItemName.
 The first to see if the value has changed, and the second to turn that value into an array.
 It would be better to change the code so that it only calls conf.get once for that value,
and then to turn it into a trimmed array of Strings call StringUtils.getTrimmedStrings, which
is what the second method does internally. 
> Deadlock in LocalDirsHandlerService and ShuffleHandler
> ------------------------------------------------------
>                 Key: MAPREDUCE-3519
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3519
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, nodemanager
>    Affects Versions: 0.23.1, 0.24.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.23.1
>         Attachments: 3519.patch, 3519.v1.patch, deadlock.txt
> MAPREDUCE-3121 cloned Configuration object in LocalDirsHandlerService.init() to avoid
others to access that configuration object. But since it is used in local FileSystem object
creation in LocalDirAllocator.AllocatorPerContext and the same FileSystem object is used in
ShuffleHandler.Shuffle.localDirAllocator, this is causing a deadlock when accessing this configuration
object from LocalDirsHandlerService and ShuffleHandler along with AllocatorPerContext object.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message