Once we settle on something, let's post it on the twiki.
On May 17, 2006, at 10:07 AM, Doug Cutting (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/HADOOP-211?
> page=comments#action_12412213 ]
>
> Doug Cutting commented on HADOOP-211:
> -------------------------------------
>
> The semantics I use for levels is something like:
>
> SEVERE: if this is a production system, someone should be paged,
> red lights should flash, etc. Something is definitely wrong and
> the system is not operating correctly. Intervention is required.
> This should be used sparingly.
>
> WARN: in a production system, warnings should be propagated &
> summarized on a central console. If lots are generated then
> something may be wrong.
>
> INFO, FINE, FINER, etc. are used for debugging. INFO is the level
> normally logged in production, FINE, FINER, etc. are typically only
> used when developing.
>
> Is that consistent with the way others use these?
>
>
>> logging improvements for Hadoop
>> -------------------------------
>>
>> Key: HADOOP-211
>> URL: http://issues.apache.org/jira/browse/HADOOP-211
>> Project: Hadoop
>> Type: Improvement
>
>> Versions: 0.2
>> Reporter: Sameer Paranjpye
>> Assignee: Sameer Paranjpye
>> Priority: Minor
>> Fix For: 0.3
>
>>
>> Here's a proposal for some impovements to the way Hadoop does
>> logging. It advocates 3
>> broad changes to the way logging is currently done, these being:
>> - The use of a uniform logging format by all Hadoop subsystems
>> - The use of Apache commons logging as a facade above an
>> underlying logging framework
>> - The use of Log4J as the underlying logging framework instead of
>> java.util.logging
>> This is largely polishing work, but it seems like it would make
>> log analysis and debugging
>> easier in the short term. In the long term, it would future proof
>> logging to the extent of
>> allowing the logging framework used to change while requiring
>> minimal code change. The
>> propos changes are motivated by the following requirements which
>> we think Hadoops
>> logging should meet:
>> - Hadoops logs should be amenable to analysis by tools like grep,
>> sed, awk etc.
>> - Log entries should be clearly annotated with a timestamp and a
>> logging level
>> - Log entries should be traceable to the subsystem from which they
>> originated
>> - The logging implementation should allow log entries to be
>> annotated with source code
>> location information like classname, methodname, file and line
>> number, without requiring
>> code changes
>> - It should be possible to change the logging implementation used
>> without having to change
>> thousands of lines of code
>> - The mapping of loggers to destinations (files, directories,
>> servers etc.) should be
>> specified and modifiable via configuration
>> Uniform logging format:
>> All Hadoop logs should have the following structure.
>> <Header>\n
>> <LogEntry>\n [<Exception>\n]
>> .
>> .
>> .
>> where the header line specifies the format of each log entry. The
>> header line has the format:
>> '# <Fieldname> <Fieldname>...\n'.
>> The default format of each log entry is: '# Timestamp Level
>> LoggerName Message', where:
>> - Timestamp is a date and time in the format MM/DD/YYYY:HH:MM:SS
>> - Level is the logging level (FATAL, WARN, DEBUG, TRACE, etc.)
>> - LoggerName is the short name of the logging subsystem from which
>> the message originated e.g.
>> fs.FSNamesystem, dfs.Datanode etc.
>> - Message is the log message produced
>> Why Apache commons logging and Log4J?
>> Apache commons logging is a facade meant to be used as a wrapper
>> around an underlying logging
>> implementation. Bridges from Apache commons logging to popular
>> logging implementations
>> (Java logging, Log4J, Avalon etc.) are implemented and available
>> as part of the commons logging
>> distribution. Implementing a bridge to an unsupported
>> implementation is fairly striaghtforward
>> and involves the implementation of subclasses of the commons
>> logging LogFactory and Logger
>> classes. Using Apache commons logging and making all logging calls
>> through it enables us to
>> move to a different logging implementation by simply changing
>> configuration in the best case.
>> Even otherwise, it incurs minimal code churn overhead.
>> Log4J offers a few benefits over java.util.logging that make it a
>> more desirable choice for the
>> logging back end.
>> - Configuration Flexibility: The mapping of loggers to
>> destinations (files, sockets etc.)
>> can be completely specified in configuration. It is possible to do
>> this with Java logging as
>> well, however, configuration is a lot more restrictive. For
>> instance, with Java logging all
>> log files must have names derived from the same pattern. For the
>> namenode, log files could
>> be named with the pattern "%h/namenode%u.log" which would put log
>> files in the user.home
>> directory with names like namenode0.log etc. With Log4J it would
>> be possible to configure
>> the namenode to emit log files with different names, say
>> heartbeats.log, namespace.log,
>> clients.log etc. Configuration variables in Log4J can also have
>> the values of system
>> properties embedded in them.
>> - Takes wrappers into account: Log4J takes into account the
>> possibility that an application
>> may be invoking it via a wrapper, such as Apache commons logging.
>> This is important because
>> logging event objects must be able to infer the context of the
>> logging call such as classname,
>> methodname etc. Inferring context is a relatively expensive
>> operation that involves creating
>> an exception and examining the stack trace to find the frame just
>> before the first frame
>> of the logging framework. It is therefore done lazily only when
>> this information actually
>> needs to be logged. Log4J can be instructed to look for the frame
>> corresponding to the wrapper
>> class, Java logging cannot. In the case of Java logging this means
>> that a) the bridge from
>> Apache commons logging is responsible for inferring the calling
>> context and setting it in the
>> logging event and b) this inference has to be done on every
>> logging call regardless of whether
>> or not it is needed.
>> - More handy features: Log4J has some handy features that Java
>> logging doesn't. A couple
>> of examples of these:
>> a) Date based rolling of log files
>> b) Format control through configuration. Log4J has a PatternLayout
>> class that can be
>> configured to generate logs with a user specified pattern. The
>> logging format described
>> above can be described as "%d{MM/dd/yyyy:HH:mm:SS} %c{2} %p %m".
>> The format specifiers
>> indicate that each log line should have the date and time followed
>> by the logger name followed
>> by the logging level or priority followed by the application
>> generated message.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the
> administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
> http://www.atlassian.com/software/jira
>
|