hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3110) NameNode does logging in critical sections just about everywhere
Date Fri, 28 Mar 2008 02:28:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582884#action_12582884
] 

Pete Wyckoff commented on HADOOP-3110:
--------------------------------------

I'm seeing 11 microseconds for a small info log message with one variable and one function
call. And about .5 microseconds for it when the logging level is error.

Maybe small here, but the hadoop synchronized methods aren't really in the spirit of critical
sections, but I'm happy to ignore this.

Of course, I don't know how long a typical call in the NN takes. If it's something like 10
milliseconds, then this is too small to matter, but if 1 or so, then this is starting to add
up.




> NameNode does logging in critical sections just about everywhere
> ----------------------------------------------------------------
>
>                 Key: HADOOP-3110
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3110
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1, 0.15.2,
0.15.3, 0.16.0, 0.16.1
>         Environment: All
>            Reporter: Pete Wyckoff
>
> e.g., FSNameSystem.addStoredBlock (but almost every method has logging in its critical
sections)
> This method is synchronized and it's spitting something out to Log.info every block stored.
Normally not a big deal, but since this is in the name node and these are critical sections...
> We shouldn't even do any logging at all in critical sections, so even the info and warn
are bad.  But, in many places in the code, it would be hard to tease these out (although eventually
they really should be), but the system could start using something like an AsyncAppender and
see how it improves performance. 
> Even though the log may have a buffer, the writing and doing the formatting and stuff
cause a drag on performance with 100s/1000s of machines trying to talk to the name node.
> At a minimum, the most often  triggered Log.info could be changed to Log.debug.
> for reference: http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/AsyncAppender.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message