On Wed, Jul 29, 2009 at 10:06 PM, Marc Limotte <mlimotte@feeva.com> wrote:
> A quick question about the user logs (hadoop 0.19.1).
>
> I've looked all over HDFS. I can't locate the actual userlogs. Where are
> they stored? I can get to them through the job tracker web interface, but
> I'd like to be able to script some jobs to check the logs for errors. I
> suppose I could use curl against the web interface, but that seems like a
> hack. Alternatively, is there command line access to the userlogs through
> the hadoop command? I couldn't find anything like that.
> Thanks for any tips...
>
>
All hadoop logs are stored on the local filesystem on each node (the
location is configurable). What you could do though is run a cron job which
periodically pushes log files to HDFS at the end of the day.
Here's another idea:
http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-for-hadoop-log-collection/
--
Harish Mallipeddi
http://blog.poundbang.in
|