hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
Date Tue, 02 Mar 2010 04:11:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840003#action_12840003

Amareshwari Sriramadasu commented on MAPREDUCE-927:

bq. Creation of the localFS should not be in a static block. In the past also, we did this
and realized it creates a kind of circular initialization of loggers and results in NPE while
creating log objects which can be seen in task-logs.
Can you elaborate this? How will static initialization code result into circular initialization
of loggers? How will this result into NPE?

bq. Also can't we simply construct a local-filesystem here itself, instead of calling TaskLog.getLocalFileSystem()?
I'm getting the log file system through TaskLog.getLogFileSystem() to make sure that it is
the same FileSystem used in TaskLog creation and deletion. 

bq. Rename the class to TaskLogsMonitor, so that we are consistent going forward with MAPREDUCE-1100.
bq. The logscleanup thread is not joined/killed in the TaskTracker.close(). So, there will
be zombie threads in the system on a re-init and may well interfere with the new thread.
TaskLogMonitor introduced in MAPREDUCE-1100 is not a daemon thread, but TaskLogCleanupThread
needs to be. I don't think we can merge the functionality of TaskLogCleanupThread and TaskLogMonitor
into one thread. 
TaskLogCleanupThread is not started after every re-init. It is started only once similar to
taskCleanupThread, directoryCleanupThread. So, this should not be joined in TaskTracker.close().

> Cleanup of task-logs should happen in TaskTracker instead of the Child
> ----------------------------------------------------------------------
>                 Key: MAPREDUCE-927
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-927
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: security, tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>            Assignee: Amareshwari Sriramadasu
>            Priority: Blocker
>             Fix For: 0.21.0
>         Attachments: patch-927-1.txt, patch-927.txt
> Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons:
1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will
get affected due to operations that TT actually should own.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message