hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3110) NameNode does logging in critical sections just about everywhere
Date Fri, 28 Mar 2008 18:44:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583136#action_12583136

Doug Cutting commented on HADOOP-3110:

> Of course, I think it is worthwhile to open a JIRA around the general issue that critical
sections are not treated as such in many places.

This seems too general for a JIRA issue.  General coding guidelines and principles are more
appropriate on the wiki, no?

The ideal performance-related JIRA describes a measurable problem.  Then specific solutions
can be evaluated.  So if, for example, you think synchronized logging is significantly affecting
namenode performance (as you seem to) then that's something we can measure and fix.  So let's
measure it first, and if it's indeed significant, let's fix it.  Since others seem to question
whether this is in fact significant, I suggested that the task of measuring it was yours,
as this would probably convince others it was worth fixing.  Does that make sense?

> NameNode does logging in critical sections just about everywhere
> ----------------------------------------------------------------
>                 Key: HADOOP-3110
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3110
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1, 0.15.2,
0.15.3, 0.16.0, 0.16.1
>         Environment: All
>            Reporter: Pete Wyckoff
> e.g., FSNameSystem.addStoredBlock (but almost every method has logging in its critical
> This method is synchronized and it's spitting something out to Log.info every block stored.
Normally not a big deal, but since this is in the name node and these are critical sections...
> We shouldn't even do any logging at all in critical sections, so even the info and warn
are bad.  But, in many places in the code, it would be hard to tease these out (although eventually
they really should be), but the system could start using something like an AsyncAppender and
see how it improves performance. 
> Even though the log may have a buffer, the writing and doing the formatting and stuff
cause a drag on performance with 100s/1000s of machines trying to talk to the name node.
> At a minimum, the most often  triggered Log.info could be changed to Log.debug.
> for reference: http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/AsyncAppender.html

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message