hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6107) Have some log messages designed for machine parsing, either real-time or post-mortem
Date Thu, 25 Jun 2009 14:22:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724104#action_12724104

Todd Lipcon commented on HADOOP-6107:

In other systems, I've done this by using Thrift to serialize the logs on disk. This allows
for backward compatible evolution of the contents of the log messages, and is a much more
compact format even without compression. We could consider Avro for this down the road.

The obvious disadvantage is that text logs are way more useful than binary ones. Having binary
logs absolutely requires that there is a very good tool for converting them to a readable
form, as well as grepping through them.

> Have some log messages designed for machine parsing, either real-time or post-mortem
> ------------------------------------------------------------------------------------
>                 Key: HADOOP-6107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6107
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
> Many programs take the log output of bits of Hadoop, and try and parse it. Some may also
put their own back end behind commons-logging, to capture the input without going via Log4J,
so as to keep the output more machine-readable.
> These programs need log messages that
> # are easy to parse by a regexp or other simple string parse  (consider quoting values,
> # push out the full exception chain rather than stringify() bits of it
> # stay stable across versions
> # log the things the tools need to analyse: events, data volumes, errors
> For these logging tools, ease of parsing, retention of data and stability over time take
the edge over readability. In HADOOP-5073, Jiaqi Tan proposed marking some of the existing
log events as evolving towards stability. As someone who regulary patches log messages to
improve diagnostics, this creates a conflict of interest. For me, good logs are ones that
help people debug their problems without anyone else helping, and if that means improving
the text, so be it. Tools like Chukwa have a different need. 
> What to do? Some options
>  # Have some messages that are designed purely for other programs to handle
>  # Have some logs specifically for machines, to which we log alongside the human-centric
>  # Fix many of the common messages, then leave them alone.
>  # Mark log messages to be left alone (somehow)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message