hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-194) Log handling in case of NM restart.
Date Wed, 10 Jul 2013 01:56:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704115#comment-13704115
] 

Bikas Saha commented on YARN-194:
---------------------------------

Why does the NM upload logs when it the container completes? It does not need to wait for
app completion. It can use HDFS append to append the logs to the same file. This is safe since
NM should be the single writer.
NM could then delete these container logs after uploading them. Risk is duplicate data whenever
NM restarts while it was in the middle of uploading a particular log.
                
> Log handling in case of NM restart.
> -----------------------------------
>
>                 Key: YARN-194
>                 URL: https://issues.apache.org/jira/browse/YARN-194
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 0.23.4
>            Reporter: Siddharth Seth
>            Assignee: Omkar Vinit Joshi
>
> Currently, if an NM restarts - existing logs will be left around till they're manually
cleaned up. The NM could be improved to handle these files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message