hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-1093) Improve namenode scalability by splitting the FSNamesystem synchronized section in a read/write lock
Date Mon, 12 Jul 2010 08:31:54 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

dhruba borthakur updated HDFS-1093:
-----------------------------------

    Attachment: NNreadwriteLock_trunk_3.txt

I have purposely kept the try-catch block un-indented so that it is easier for you to review.
Once the review is complete, I will upload a final patch with proper indentation.

FSNamesystem.java
-------------------------
I removed the import of the ReadWriteLock. However, I kept the fsLock initializing inside
the initialize method rather than moving it to the declaration. Most variables seem to be
getting initialized inside the initialize() method, isn't it?

getBlockLocations() - I now acquire the readLock and attempt to proceed ahead. If the access-time
has to be set, then I release the readLock, acquire the writeLock and start all over again.
Check for safemode should be inside the FSnamesystem lock, in fact more of it is coming in
HDFS-988.
setTimes() move isAccessTimeSupported() call out side the lock : done.

CommitBlockSynchronization(): the key is to call the logSync() outside the FSnamesystem lock.
And there is a LOG statement that prints out "src" after the call to logSync(). That is the
reason why i had to declare "String src" right at the beginning of the method.

I agree with you that there are code portions in processReport, createSymLinkInternal, startFileInternal()
that  can move outside the FSNamesystem lock. However, I would like to avoid doing this code
reorganizatin as part of this JIRA, especially because it makes the code difficult to review.
Also, this is not a regression because the original code has all these code inside the synchronized
section anyway. Please let me know if you agree on this one.
getDatanodeListForReport - move boolean and HashMap out of read lock: done.
getSafeModeTip - readLock instead of writeLock : done

BlockManager.java
------------------------
chooseUnderReplicatedBlocks - should we grab readLock instead of writeLock? I wold prefer
to keep the writeLock because this method updates member variables of the class, e.g. missingBlocksInPrevIter

removeStoredBlock - assert to be replaced by grab writeLock: I have the impression that all
calls to removeStoredBlock already has the writeLock, that is the reason for the assert. Do
you know of a code path via which this is not the case?

DecommissionManager.java - readLock instead of writeLock; The DecommissionManager.check()
calls FSNamesystem.checkDecommissionStateInterna() which, in turn, invokes node.setDecommissioned().
This updates the state of the node, hence I was confortable keeping the writeLock in this
code path.

FSDirectory.java
---------------------
 BackupStorage.loadCheckPoint() is now fixed.
Do you have a suggestion on how I can fix the code in FSPermissionChecker.checkPermission()?


> Improve namenode scalability by splitting the FSNamesystem synchronized section in a
read/write lock
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1093
>                 URL: https://issues.apache.org/jira/browse/HDFS-1093
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: NNreadwriteLock.txt, NNreadwriteLock_trunk_1.txt, NNreadwriteLock_trunk_2.txt,
NNreadwriteLock_trunk_3.txt
>
>
> Most critical data structures in the NameNode (NN) are protected by a syncronized methods
in the FSNamesystem class. This essentially makes critical code paths in the NN single-threaded.
However, a large percentage of the NN calls are listStatus, getBlockLocations, etc which do
not change internal data structures at all, these are read-only calls. If we change the FSNamesystem
lock to a read/write lock, many of the above operations can occur in parallel, thus improving
the scalability of the NN.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message