hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tomasz Nykiel (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2495) Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock
Date Mon, 24 Oct 2011 10:01:32 GMT
Increase granularity of write operations in ReplicationMonitor thus reducing contention for
write lock
------------------------------------------------------------------------------------------------------

                 Key: HDFS-2495
                 URL: https://issues.apache.org/jira/browse/HDFS-2495
             Project: Hadoop HDFS
          Issue Type: Sub-task
            Reporter: Tomasz Nykiel
            Assignee: Tomasz Nykiel


For processing blocks in ReplicationMonitor (BlockManager.computeReplicationWork), we first
obtain a list of blocks to be replicated by calling chooseUnderReplicatedBlocks, and then
for each block which was found, we call computeReplicationWorkForBlock. The latter processes
a block in three stages, acquiring the writelock twice per call:

1. obtaining block related info (livenodes, srcnode, etc.) under lock
2. choosing target for replication
3. scheduling replication (under lock)

We would like to change this behaviour and decrease contention for the write lock, by batching
blocks and executing 1,2,3, for sets of blocks, rather than for each one separately. This
would decrease the number of writeLock to 2, from 2*numberofblocks.

Also, the info level logging can be pushed outside the writelock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message