hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1648) Use rolling to limit tasklogs
Date Wed, 21 Apr 2010 10:56:50 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859296#action_12859296
] 

Vinod K V commented on MAPREDUCE-1648:
--------------------------------------

Guilin, I opened MAPREDUCE-1648 as a sub-task for MAPREDUCE-1100. Given some uncertainities
with your patch w.r.t performance and pipes, I am using MAPREDUCE-1648 as an incremental imporovement
over what we have at present for task logs to address the pressing concern of excessive logging.
Please comment if you wish or raise any objections there. Thanks!

> Use rolling to limit tasklogs
> -----------------------------
>
>                 Key: MAPREDUCE-1648
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1648
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>            Reporter: Guilin Sun
>            Priority: Minor
>         Attachments: syslog.patch
>
>
> There are at least two types of task-logs: syslog and stdlog
> Task-Jvm outputs syslog by log4j with TaskLogAppender, TaskLogAppender looks just like
"tail -c", it stores last N byte/line logs in memory(via queue), and do real output only if
all logs is commit and Appender is going to close.
> The common problem of TaskLogAppender and 'tail -c'  is keep everything in memory and
user can't see any log output while task is in progress.
> So I'm going to try RollingFileAppender  instead of  TaskLogAppender, use MaxFileSize&MaxBackupIndex
to limit log file size.
> RollingFileAppender is also suitable for stdout/stderr, just redirect stdout/stderr to
log4j via LoggingOutputStream, no client code have to be changed, and RollingFileAppender
seems better than 'tail -c' too.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message