hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Han (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
Date Thu, 12 Sep 2013 20:35:54 GMT

    [ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765870#comment-13765870
] 

Paul Han commented on YARN-1130:
--------------------------------

Patch is submitted. 
A few notes here:
# Added a LogUtils class to facilitate triggering log flush since Log4J's interface such as
LogManager doesn't provide a flush() interface
# Modified Task class to trigger the flushing of logs before it sends TASK_DONE event to MRMaster.
In some cases, TASK_DONE may trigger the container to be killed before all log being written
to disk
# ContainerLogAppender supports to flush the log when a special message is received. The log
flush is done in a "synchronous" manner with a timeout. This will ensure the invoker of flush
waits until logs are written to disk or timeout happens.
                
> Improve the log flushing for tasks when mapred.userlog.limit.kb is set
> ----------------------------------------------------------------------
>
>                 Key: YARN-1130
>                 URL: https://issues.apache.org/jira/browse/YARN-1130
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.0.5-alpha
>            Reporter: Paul Han
>             Fix For: 2.0.5-alpha
>
>         Attachments: YARN-1130.patch
>
>
> When userlog limit is set with something like this:
> {code}
> <property>
> <name>mapred.userlog.limit.kb</name>
> <value>2048</value>
> <description>The maximum size of user-logs of each task in KB. 0 disables the cap.
> </description>
> </property>
> {code}
> the log entry will be truncated randomly for the jobs.
> The log size is left between 1.2MB to 1.6MB.
> Since the log is already limited, avoid the log truncation is crucial for user.
> The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is
that log entries will not flush to file until the container shutdown and logmanager close
all appenders. If user likes to see the log during task execution, it doesn't support it.
> Will propose a patch to add a flush mechanism and also flush the log when task is done.
 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message