hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
Date Thu, 01 Apr 2010 16:18:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852434#action_12852434
] 

Ravi Gummadi commented on MAPREDUCE-1100:
-----------------------------------------

The patch existing now at MAPREDUCE-1057 doesn't solve the whole problem. Currently, "tail
-c" is done after task process has finished execution but log.index file is created(and written
to) by task itself. So log.index cannot have correct index details of task logs(i.e. startingOffset
and length of stdout, stderr and syslog).

One way to solve this is to make TT write the index details to log.index file once a task
is done.

Thoughts ?

> User's task-logs filling up local disks on the TaskTrackers
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-1100
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: MAPREDUCE-1100-20091102.txt, MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt,
patch-1100-fix-ydist.2.txt
>
>
> Some user's jobs are filling up TT disks by outrageous logging. mapreduce.task.userlog.limit.kb
is not enabled on the cluster. Disks are getting filled up before task-log cleanup via mapred.task.userlog.retain.hours
can kick in.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message