Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Mon, 13 Jun 2011 20:22:51 +0000 (UTC)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: 
 <1533906232.3739.1307996571642.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1441496444.3456.1307991051510.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (MAPREDUCE-2589) TaskTracker not purging userlog
 directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Crawford updated MAPREDUCE-2589:
---------------------------------------

    Attachment: cleanup_userlogs.py

We see this on our clusters too.

Attached is a script that I run from cron to cleanup old userlogs. The general idea is setting some high water mark for userlog disk space, and when passed, delete logs until passing some low water mark. Logs for running jobs are excluded from cleanup, which has infrequently caused issues but in general are worth excluding.

Posting as an example of what the replacement might look like (as an internal periodic task, of course). Also, not sure how the nextgen stuff deals with cleanup.

> TaskTracker not purging userlog directories
> -------------------------------------------
>
>                 Key: MAPREDUCE-2589
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.205.0
>         Environment: 0.20.205
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>            Priority: Minor
>         Attachments: cleanup_userlogs.py
>
>
> UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually
> cleaned. Things can accumulate over a period of time.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira