hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuan Gong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5237) Not all logs get aggregated with rolling log aggregation.
Date Sat, 11 Jun 2016 00:09:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15325539#comment-15325539

Xuan Gong commented on YARN-5237:

Currently, we remove the finished container when we do the log aggregation which works fine
if rolling log aggregation is disabled. Because the log aggregation only happens once. 

But, it would cause the issue if rolling log aggregation is enabled. Because, if we set includePattern/excludePattern,
and we would not consider this during the application is still running. If the container finishes,
and we remove the finished container, we would lose the chance to do the log aggregation (for
includePattern/excludePattern) when the app finishes.

Set<ContainerId> finishedContainers =
    new HashSet<ContainerId>(pendingContainerInThisCycle);

// This container is finished, and all its logs have been uploaded,
// remove it from containerLogAggregators.
if (finishedContainers.contains(container)) {

> Not all logs get aggregated with rolling log aggregation.
> ---------------------------------------------------------
>                 Key: YARN-5237
>                 URL: https://issues.apache.org/jira/browse/YARN-5237
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-5237.1.patch
> Steps to reproduce:
> 1) enable RM recovery
> 2) Run a sleep job
> 3) restart RM
> 4) kill the application
> We can not find that the logs for the first attempt

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message