hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
Date Tue, 09 Feb 2010 11:23:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831417#action_12831417
] 

Amareshwari Sriramadasu commented on MAPREDUCE-927:
---------------------------------------------------

With the current proposal, we found two things that need an answer.
# Memory footprint of the TaskTracker: Each map entry (JobID, Long) would take about 40 bytes.
If the userLogRetainsHours is configured to 7days and there are 1lakh job's tasks run by a
TaskTracker in a day, the map would take up 28MB of memory. I guess this memory footprint
is fine compared to persisting the same information to disk and reading it back and forth
from disk until the directory is removed.
# If TaskTracker is reinited/ restarted and a job completed when the TaskTracker was down,
then TaskTracker would not get a KillJobAction for the job. Then we can keep the userlogs
for default userLogRetainsHours, after the reinit/restart.

Thoughts?

> Cleanup of task-logs should happen in TaskTracker instead of the Child
> ----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-927
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-927
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: security, tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>            Assignee: Amareshwari Sriramadasu
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons:
1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will
get affected due to operations that TT actually should own.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message