hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amir Langer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5453) Support fine grain locking in FSNamesystem
Date Tue, 24 Dec 2013 10:12:50 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13856257#comment-13856257

Amir Langer commented on HDFS-5453:

Hi [~sureshms]  

I worked on the async simulation with [~ebortnik] and can further explain the results. 
First the caveats / general comments:
*   It is a simulation, not an implementation. The objective was just to get a rough estimate
of how much potential is there in having such an approach.
*   We realise an async approach will also introduce new issues which are avoided by the current
architecture. This is a purely theoretical idea at this stage.
*   As Eddie said - It is orthogonal to this issue. We might want to move this discussion
elsewhere (or in other words... apologies to anyone who was only interested in fine grained
locking and have read so far :) ) 

As for the results:

BASE is the existing locking mechanism.
ASYNC is the same code without any lock or synchronisation but with added queue and a scheduler
thread that polls requests and assigns those to worker threads. Before assigning it, the scheduler
thread also resolves the path for every request (as we assume this should form part of the
decision to which worker to assign the request - This is where it is somewhat similar to the
fine grained locking idea).
Edit logs are not written in the ASYNC simulation - The idea is to create a radically different
architecture where the requests are Asynchronous, the edit log would be the persistent representation
of the work queue and only a client adapter would turn this into a synchronous API. (Again,
above caveats apply here)
The benchmark bypasses the RPC layer and measures the namespace alone. 
The number of clients/workers represents the number of threads in BASE (Number of clients
accessing the namespace concurrently).
In ASYNC it is the number of worker threads and also the number of threads acting as clients
(Feeding the queue with requests). Remember - ASYNC also has an extra thread acting as scheduler.

As for the GET_BLOCK_LOCATIONS results - It does benefit from the read lock to an extent (notice
a steady rise in throughput up to 4 clients which is half the number of cores we used). 
I believe it does not benefit fully from it due to two reasons:

#   Other synchronisation points which synchronises all threads like - 
#*   FSPermissionChecker uses several bits of synchronised code especially org.apache.hadoop.security.UserGroupInformation#getCurrentUser
which seems to frequently block threads (This may be different in other setups but that's
how our benchmark works).
#*   audit logs
#*   DNSToSwitchMapping#resolve drills down to synchronised code in InetAddress
#   Several Background threads which periodically grab a write lock and block all other threads

I'll be happy to discuss further / attach a patch but perhaps this is too off-topic for this
specific JIRA.

> Support fine grain locking in FSNamesystem
> ------------------------------------------
>                 Key: HDFS-5453
>                 URL: https://issues.apache.org/jira/browse/HDFS-5453
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode
>    Affects Versions: 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: async_simulation.xlsx
> The namesystem currently uses a course grain lock to control access.  This prevents concurrent
writers in different branches of the tree, and prevents readers from accessing branches that
writers aren't using.
> Features that introduce latency to namesystem operations, such as cold storage of inodes,
will need fine grain locking to avoid degrading the entire namesystem's throughput.

This message was sent by Atlassian JIRA

View raw message