hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5453) Support fine grain locking in FSNamesystem
Date Mon, 30 Dec 2013 00:11:50 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13858491#comment-13858491

Konstantin Shvachko commented on HDFS-5453:

It is indeed strange as Suresh mentioned, that get block locations performance in ASYNC falls
way below BASE with 15 and 16 threads. Not asking what happens with 200 threads only because
you refer to it as a simulation with  a radically different architecture, which should probably
be discussed elsewhere. 
BTW, for the BASE case you should see improvement with 200 handlers as it starts benefiting
from edits batching.

As for fine grain locking it looks to me that your main motivation is not the performance
but rather the new functionality pursued with HDFS-5477. BTW should it be linked here then.
I see you propose to separate BM and access it with rpc calls from FSNamesystem. But I don't
quite understand the consistency model. With fine or coarse locking you will hold a lock while
making an rpc call. If the rpc fails you will need to provide consistency of the namespace
state and the retries with e.g. at-most-once semantics, mentioned in the design. So why not
just release the lock while making the rpc and reacquire it when the rpc is completed. In
a sense the lock here is a flag signalling a BM call is in progress so that others could avoid
accessing the INode.
I mean one way or another a consistency logic should be in place to handle failures. And may
be there is a way to design the feature without relying on locking to provide consistency
in general.

> Support fine grain locking in FSNamesystem
> ------------------------------------------
>                 Key: HDFS-5453
>                 URL: https://issues.apache.org/jira/browse/HDFS-5453
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode
>    Affects Versions: 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: async_simulation.xlsx
> The namesystem currently uses a course grain lock to control access.  This prevents concurrent
writers in different branches of the tree, and prevents readers from accessing branches that
writers aren't using.
> Features that introduce latency to namesystem operations, such as cold storage of inodes,
will need fine grain locking to avoid degrading the entire namesystem's throughput.

This message was sent by Atlassian JIRA

View raw message