hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1181) userlogs reader
Date Tue, 03 Apr 2007 18:04:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486445
] 

Doug Cutting commented on HADOOP-1181:
--------------------------------------

> Actually (we probably should take this discussion elsewhere, so +1 for a separate issue)

Done.  This is now HADOOP-1199.

> the idea of starting another mapred job to get the logs of a previous mapred job doesn't
seem right ... 

It's not to get the logs, but to process the logs and get summary statistics, like the most
frequent warnings logged.  We shouldn't force folks to copy logs to DFS to determine that.


> userlogs reader
> ---------------
>
>                 Key: HADOOP-1181
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1181
>             Project: Hadoop
>          Issue Type: Improvement
>            Reporter: stack@archive.org
>         Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging.  I want to be able to quickly parse the logs across the
cluster for anomalies.  org.apache.hadoop.tool.Logalyzer looks promising at first but it does
not know how to deal with the userlog format  and it wants to first copy all logs local. 
Digging, there does not seem to currently be a reader for hadoop userlog format.  TaskLog$Reader
is not generally accessible and it too expects logs to be on the local filesystem (The latter
is of little good if I want to run the analysis as a mapreduce job).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message