Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 8 Nov 2011 13:27:54 +0000 (UTC)
From: "Hudson (Commented) (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <346164995.10757.1320758874135.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <395318687.8074.1319450492655.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HDFS-2495) Increase granularity of write
 operations in ReplicationMonitor thus reducing contention for write lock
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146301#comment-13146301 ] 

Hudson commented on HDFS-2495:
------------------------------

Integrated in Hadoop-Mapreduce-trunk #891 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/891/])
    HDFS-2495. Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock. Contributed by Tomasz Nykiel.

hairong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199024
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java

                
> Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2495
>                 URL: https://issues.apache.org/jira/browse/HDFS-2495
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: name-node
>            Reporter: Tomasz Nykiel
>            Assignee: Tomasz Nykiel
>             Fix For: 0.24.0
>
>         Attachments: replicationMon.patch, replicationMon.patch-1
>
>
> For processing blocks in ReplicationMonitor (BlockManager.computeReplicationWork), we first obtain a list of blocks to be replicated by calling chooseUnderReplicatedBlocks, and then for each block which was found, we call computeReplicationWorkForBlock. The latter processes a block in three stages, acquiring the writelock twice per call:
> 1. obtaining block related info (livenodes, srcnode, etc.) under lock
> 2. choosing target for replication
> 3. scheduling replication (under lock)
> We would like to change this behaviour and decrease contention for the write lock, by batching blocks and executing 1,2,3, for sets of blocks, rather than for each one separately. This would decrease the number of writeLock to 2, from 2*numberofblocks.
> Also, the info level logging can be pushed outside the writelock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira