hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4080) Add a separate logger for block state change logs to enable turning off those logs
Date Sat, 10 Nov 2012 10:51:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494614#comment-13494614
] 

Hudson commented on HDFS-4080:
------------------------------

Integrated in Hadoop-Yarn-trunk #32 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/32/])
    Moving HDFS-4080 to Incompatible Change section. (Revision 1407567)
HDFS-4080. Add a separate logger for block state change logs to enable turning off those logs.
Contributed by Kihwal Lee. (Revision 1407566)

     Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407567
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407566
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeDeath.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend2.java

                
> Add a separate logger for block state change logs to enable turning off those logs
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-4080
>                 URL: https://issues.apache.org/jira/browse/HDFS-4080
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.3-alpha
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>             Fix For: 3.0.0, 2.0.3-alpha, 0.23.5
>
>         Attachments: hdfs-4080.1.patch, hdfs-4080-branch-0.23.patch, hdfs-4080.patch,
hdfs-4080.patch, hdfs-4080-trunk.patch
>
>
> Although the block-level logging in namenode is useful for debugging, it can add a significant
overhead to busy hdfs clusters since they are done while the namespace write lock is held.
One example is shown in HDFS-4075. In this example, the write lock was held for 5 minutes
while logging 11 million log messages for 5.5 million block invalidation events. 
> It will be useful if we have an option to disable these block-level log messages and
keep other state change messages going.  If others feel that they can turned into DEBUG (with
addition of isDebugEnabled() checks), that may also work too, but there might be people depending
on the messages.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message