hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-171) NodeManager should serve logs directly if log-aggregation is not enabled
Date Fri, 19 Oct 2012 08:26:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479880#comment-13479880

Siddharth Seth commented on YARN-171:

Ignore the last comment. At least the bit about the NM not serving logs after App completion.
When aggregation is disabled, the NM is keeping track of applications till the logs are active
on the NM - which sets it up nicely for an OOM on busy clusters, depending on the log-retention
Accessing the logs requires some application info - ACLs, owner. Maintaining this in memory
for the log-retention time could be problematic. One option could be to create an index similar
to the history server on local FS. Would prefer not creating a single file per app though.
Any suggestions on a lightweight key value store with a compatible license ?
> NodeManager should serve logs directly if log-aggregation is not enabled
> ------------------------------------------------------------------------
>                 Key: YARN-171
>                 URL: https://issues.apache.org/jira/browse/YARN-171
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 0.23.3
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Siddharth Seth
> NodeManagers never serve logs for completed applications. If log-aggregation is not enabled,
in the interim, due to bugs like YARN-162, this is a serious problem for users as logs are
necessarily not available.
> We should let nodes serve logs directly if YarnConfiguration.LOG_AGGREGATION_ENABLED
is set. This should be okay as NonAggregatingLogHandler can retain logs upto YarnConfiguration.NM_LOG_RETAIN_SECONDS.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message