hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8149) cap space usage of default log4j rolling policy
Date Thu, 22 Mar 2012 22:18:22 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236101#comment-13236101
] 

Eli Collins commented on HADOOP-8149:
-------------------------------------

Pat and I sync up wrt Kihwal's point about HADOOP_NAMENODE_OPTS and his bullet #2 above. 

- HADOOP_NAMENODE_OPTS, HADOOP_SECONDARYNAMENODE_OPTS, and HADOOP_DATANODE_OPTS all hard-code
the logger values and instead use an environment variable. The NN and the 2NN are the same
and can use HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER instead of needing new service specific
variables (this is how yarn works).
- HADOOP_DATANODE_OPTS uses ERROR,DRFAS instead of INFO,DRFAS like the NN and 2NN. It was
originally created this way, anyone know why? Seems like it should use the same setting as
the NN and 2NN and be exported like them as well. If that the case then we can use HADOOP_ROOT/SECURITY_LOGGER
here as well.
- Filed HADOOP-8200 with a patch to remove HADOOP_JOBTRACKER/TASKTRACKER_OPTS which are no
longer relevant on trunk/23 so we can ignore these here

                
> cap space usage of default log4j rolling policy 
> ------------------------------------------------
>
>                 Key: HADOOP-8149
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8149
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Patrick Hunt
>            Assignee: Patrick Hunt
>         Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed
after some time and accumulate. Changes to Hadoop's default log4j file appender would help
with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization
issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however
increase reliability.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message