hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Travis Crawford (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-2589) TaskTracker not purging userlog directories
Date Mon, 13 Jun 2011 20:22:51 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Travis Crawford updated MAPREDUCE-2589:

    Attachment: cleanup_userlogs.py

We see this on our clusters too.

Attached is a script that I run from cron to cleanup old userlogs. The general idea is setting
some high water mark for userlog disk space, and when passed, delete logs until passing some
low water mark. Logs for running jobs are excluded from cleanup, which has infrequently caused
issues but in general are worth excluding.

Posting as an example of what the replacement might look like (as an internal periodic task,
of course). Also, not sure how the nextgen stuff deals with cleanup.

> TaskTracker not purging userlog directories
> -------------------------------------------
>                 Key: MAPREDUCE-2589
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions:
>         Environment: 0.20.205
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>            Priority: Minor
>         Attachments: cleanup_userlogs.py
> UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be
> cleaned. Things can accumulate over a period of time.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message