Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1117799329.1206727944454.JavaMail.jira@brutus>
Date: Fri, 28 Mar 2008 11:12:24 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3110) NameNode does logging in critical
 sections just about everywhere
In-Reply-To: <1469908286.1206652524431.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583120#action_12583120 ] 

Doug Cutting commented on HADOOP-3110:
--------------------------------------

Pete, as the reporter of this issue, I think the onus is on you to demonstrate that this is causing problems and is not just a theoretical issue.  Raghu suggested disabling logging and seeing if things perform better.  Or better yet, you could figure out how to configure AsyncAppender and provide a patch for this.

> NameNode does logging in critical sections just about everywhere
> ----------------------------------------------------------------
>
>                 Key: HADOOP-3110
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3110
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1, 0.15.2, 0.15.3, 0.16.0, 0.16.1
>         Environment: All
>            Reporter: Pete Wyckoff
>
> e.g., FSNameSystem.addStoredBlock (but almost every method has logging in its critical sections)
> This method is synchronized and it's spitting something out to Log.info every block stored. Normally not a big deal, but since this is in the name node and these are critical sections...
> We shouldn't even do any logging at all in critical sections, so even the info and warn are bad.  But, in many places in the code, it would be hard to tease these out (although eventually they really should be), but the system could start using something like an AsyncAppender and see how it improves performance. 
> Even though the log may have a buffer, the writing and doing the formatting and stuff cause a drag on performance with 100s/1000s of machines trying to talk to the name node.
> At a minimum, the most often  triggered Log.info could be changed to Log.debug.
> for reference: http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/AsyncAppender.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.