hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
Date Tue, 09 Feb 2010 11:23:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831417#action_12831417

Amareshwari Sriramadasu commented on MAPREDUCE-927:

With the current proposal, we found two things that need an answer.
# Memory footprint of the TaskTracker: Each map entry (JobID, Long) would take about 40 bytes.
If the userLogRetainsHours is configured to 7days and there are 1lakh job's tasks run by a
TaskTracker in a day, the map would take up 28MB of memory. I guess this memory footprint
is fine compared to persisting the same information to disk and reading it back and forth
from disk until the directory is removed.
# If TaskTracker is reinited/ restarted and a job completed when the TaskTracker was down,
then TaskTracker would not get a KillJobAction for the job. Then we can keep the userlogs
for default userLogRetainsHours, after the reinit/restart.


> Cleanup of task-logs should happen in TaskTracker instead of the Child
> ----------------------------------------------------------------------
>                 Key: MAPREDUCE-927
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-927
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: security, tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>            Assignee: Amareshwari Sriramadasu
>            Priority: Blocker
>             Fix For: 0.21.0
> Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons:
1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will
get affected due to operations that TT actually should own.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message