hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5453) Support fine grain locking in FSNamesystem
Date Thu, 14 Nov 2013 17:01:30 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822613#comment-13822613

Daryn Sharp commented on HDFS-5453:

I've got an implementation of fine grain fsn locking that can be phased in with no compatibility
issue.  The changes to fsn are absolutely minimal.  The use of course (existing) or fine grain
locking can be controlled via a conf.  If we clearly mark the conf as a non-public experimental
feature, then no risk will be introduced.

However...  Although readers and writers can enter disjoint branches of the namespace, the
fsdir lock reserializes batches of readers with writers.  The fsdir lock is being used to
protect various non-thread safe subsystems and data structures.  Changes to eliminate this
bottleneck may require a feature branch if addressing the bottlenecks proves too risky.  I've
been juggling too many tasks to gauge this complexity.

> Support fine grain locking in FSNamesystem
> ------------------------------------------
>                 Key: HDFS-5453
>                 URL: https://issues.apache.org/jira/browse/HDFS-5453
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode
>    Affects Versions: 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
> The namesystem currently uses a course grain lock to control access.  This prevents concurrent
writers in different branches of the tree, and prevents readers from accessing branches that
writers aren't using.
> Features that introduce latency to namesystem operations, such as cold storage of inodes,
will need fine grain locking to avoid degrading the entire namesystem's throughput.

This message was sent by Atlassian JIRA

View raw message