hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
Date Wed, 14 Oct 2009 10:30:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765503#action_12765503
] 

Vinod K V commented on MAPREDUCE-1100:
--------------------------------------

Offline discussions with Koji/Hemanth/Amareshwari/Jothi yielded a few possible approaches
to the problem in general.

The problem can be spilt into two -

h4. Shielding against a single task writing excessive logs
This can be done in various ways
 - Each task's log file entitled to a limit similar to mapred.userlog.limit.kb and the corresponding
task will be killed when the size of a log file goes over the limit.
    -- Requirements (1) and (2) are met.
    -- A bit aggressive.
 - Let the tasks run as they are now, but only keep the last mapred.userlog.limit.kb of each
log file's data when tasks finish.
    -- (2) can be met.
    -- (1) is not met. It can still fill up disks in the interim.
    -- Not aggressive.
 - Monitor the total usage of all logs and make sure that the total usage is limited. When
ever the limit can be potentially overflown, cleaning up logs by picking up tasks - say the
largest log files or the least recently ones written to.
  -- This may also remove logs of running tasks and so (2) will not be met sometimes - removing
task log files of running tasks will fail them.
  -- If running tasks' logs are not to be removed, (1) will not be met.

The first solution seems the best of the lot. Thoughts?

h4. Addressing tasks filling up disks with logs over time though each task writes only a limited
amount
This involves two different steps.
 - There should be a configurable upper limit on total disk space usable by task logs. Whenever
this limit is breached, the logs of picked up tasks should be purged.
    -- Requirement (4) will not be met, but we can hardly help.
    -- Tasks  picked up can be based on some criterion:
      --- the least recently written to
      --- logs occupying largest disk space
 - A thread running periodically should use _mapred.task.userlog.retain.hours_ to purge logs
of a particular task to avoid the total usage filling up disks.

> User's task-logs filling up local disks on the TaskTrackers
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-1100
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>
> Some user's jobs are filling up TT disks by outrageous logging. mapreduce.task.userlog.limit.kb
is not enabled on the cluster. Disks are getting filled up before task-log cleanup via mapred.task.userlog.retain.hours
can kick in.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message