hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8149) cap space usage of default log4j rolling policy
Date Wed, 14 Mar 2012 17:50:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229428#comment-13229428
] 

Eli Collins commented on HADOOP-8149:
-------------------------------------

bq. Have you noticed cases of data loss issues. Because I have not heard of these instances.

No, me neither. We do see the issue with partitions filling up frequently, people expect this
to work out of the box.

bq. They should set the logger to RFA only when HADOOP_*_LOGGER is not already set

Sounds like a plan. Pat, how about updating the patch to (a) add back the DRFA log4j.properties
section and (b) on set HADOOP_*_LOGGER if they're already unset so they can be overriden in
hadoop-env.sh?
                
> cap space usage of default log4j rolling policy 
> ------------------------------------------------
>
>                 Key: HADOOP-8149
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8149
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Patrick Hunt
>            Assignee: Patrick Hunt
>         Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed
after some time and accumulate. Changes to Hadoop's default log4j file appender would help
with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization
issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however
increase reliability.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message