hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-5664) Use of ReentrantLock.lock() in MapOutputBuffer takes up too much cpu time
Date Fri, 24 Apr 2009 07:36:30 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Douglas updated HADOOP-5664:

    Attachment: 5664-0.patch

Discussing this with Owen, he had an idea that has all the properties of the extent design
without requiring a major rewrite. Since the collection thread can determine when it next
needs to validate the state of the buffer, it can freely write up to that boundary.

Attached is a preliminary patch that removes not only some of the excessive spill synchronization,
but it also drops the synchronization on MapOutputBuffer.Buffer methods. Since the buffer
is only really protected by the lock on collect, synchronizing the buffer methods can only
make potential corruption well-ordered.

Bryan: would you mind testing this with your job? I'm still working through its correctness
(it passes unit tests, at least), but I'm curious about its effect in your environment.

> Use of ReentrantLock.lock() in MapOutputBuffer takes up too much cpu time
> -------------------------------------------------------------------------
>                 Key: HADOOP-5664
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5664
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.1
>            Reporter: Bryan Duxbury
>            Assignee: Chris Douglas
>            Priority: Minor
>         Attachments: 5664-0.patch
> In examining a profile of one of my mappers today, I noticed that the method ReentrantLock.lock()
in MapTask$MapOutputBuffer seems to be taking up ~11 seconds out of around 100 seconds total.
It seems like 10% is an awfully large amount of time to spend in this lock. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message