hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Hunt (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8149) cap space usage of default log4j rolling policy
Date Fri, 16 Mar 2012 17:41:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231421#comment-13231421

Patrick Hunt commented on HADOOP-8149:

Seems to be some consensus around 5gb. I'll update the patch to use max index of 20, with
max file size of 256mb.

@kihwal - You are specifically referring to this line, correct?

bq. export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,RFAAUDIT

where setting the security logger is overridden.

I was hesitant to change hadoop-env.sh for a couple reasons, appreciate your help on this.

1) my understanding is that hadoop-env.sh is customized by the end user. They would edit this
file with specific requirements/customizations, but that the default should handle the common

2) notice the following four lines, they are a hodgepodge of inconsistent threshold/appender
settings. Are you suggesting that we override all of these based on the HADOOP_SECURITY_LOGGER

Esp wrt item 2) I was thinking along the lines of your other comment (re scope) - that addressing
this would be good to break off into a separate jira focused exclusively on that issue. However
perhaps I'm mistaken and there's a simple solution here that I'm missing.
> cap space usage of default log4j rolling policy 
> ------------------------------------------------
>                 Key: HADOOP-8149
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8149
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Patrick Hunt
>            Assignee: Patrick Hunt
>         Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
> I've seen several critical production issues because logs are not automatically removed
after some time and accumulate. Changes to Hadoop's default log4j file appender would help
with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization
issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however
increase reliability.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message