hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2495) Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock
Date Mon, 07 Nov 2011 23:22:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145931#comment-13145931
] 

Hadoop QA commented on HDFS-2495:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502568/replicationMon.patch-1
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit
warnings.

    -1 core tests.  The patch failed these unit tests:
                  org.apache.hadoop.hdfs.TestFileAppend2
                  org.apache.hadoop.hdfs.TestBalancerBandwidth

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1540//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1540//console

This message is automatically generated.
                
> Increase granularity of write operations in ReplicationMonitor thus reducing contention
for write lock
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2495
>                 URL: https://issues.apache.org/jira/browse/HDFS-2495
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: name-node
>            Reporter: Tomasz Nykiel
>            Assignee: Tomasz Nykiel
>         Attachments: replicationMon.patch, replicationMon.patch-1
>
>
> For processing blocks in ReplicationMonitor (BlockManager.computeReplicationWork), we
first obtain a list of blocks to be replicated by calling chooseUnderReplicatedBlocks, and
then for each block which was found, we call computeReplicationWorkForBlock. The latter processes
a block in three stages, acquiring the writelock twice per call:
> 1. obtaining block related info (livenodes, srcnode, etc.) under lock
> 2. choosing target for replication
> 3. scheduling replication (under lock)
> We would like to change this behaviour and decrease contention for the write lock, by
batching blocks and executing 1,2,3, for sets of blocks, rather than for each one separately.
This would decrease the number of writeLock to 2, from 2*numberofblocks.
> Also, the info level logging can be pushed outside the writelock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message