hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10048) LocalDirAllocator should avoid holding locks while accessing the filesystem
Date Mon, 20 Jun 2016 16:03:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339794#comment-15339794
] 

Jason Lowe commented on HADOOP-10048:
-------------------------------------

Technically this is a performance optimization improvement rather than a bug fix. Typically
we wouldn't backport since those other releases are in maintenance and should only receive
fixes rather than features/improvements to reduce the risk of destabilizing those releases.
 If we feel this is an important enough performance improvement and the rewards outweigh the
risks then I'm OK with it.  It doesn't pick cleanly, but it would if we also backported HADOOP-8436,
HADOOP-8437, and HADOOP-12252 which are all bug fixes.


> LocalDirAllocator should avoid holding locks while accessing the filesystem
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-10048
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10048
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 2.3.0
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>             Fix For: 2.8.0
>
>         Attachments: HADOOP-10048.003.patch, HADOOP-10048.004.patch, HADOOP-10048.005.patch,
HADOOP-10048.006.patch, HADOOP-10048.patch, HADOOP-10048.trunk.patch
>
>
> As noted in MAPREDUCE-5584 and HADOOP-7016, LocalDirAllocator can be a bottleneck for
multithreaded setups like the ShuffleHandler.  We should consider moving to a lockless design
or minimizing the critical sections to a very small amount of time that does not involve I/O
operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message