hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5664) Use of ReentrantLock.lock() in MapOutputBuffer takes up too much cpu time
Date Mon, 13 Apr 2009 23:10:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698582#action_12698582

Chris Douglas commented on HADOOP-5664:

Such small writes are definitely a worst-case for the buffer, but I agree: 10% is steep. If
the buffer could query the serialized size of the key and value then the locking could be
relaxed somewhat, but given its current semantics I don't see a clear way to dispose of the
lock acquisition for each write without introducing additional copies.

How invasive is the profiling? Is it possible that the overhead in tracing the call to lock()
is responsible for some of the 11s overhead? Is the collection thread ever blocked waiting
for the spill to complete?

> Use of ReentrantLock.lock() in MapOutputBuffer takes up too much cpu time
> -------------------------------------------------------------------------
>                 Key: HADOOP-5664
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5664
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.1
>            Reporter: Bryan Duxbury
>            Priority: Minor
> In examining a profile of one of my mappers today, I noticed that the method ReentrantLock.lock()
in MapTask$MapOutputBuffer seems to be taking up ~11 seconds out of around 100 seconds total.
It seems like 10% is an awfully large amount of time to spend in this lock. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message