hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3476) Nodemanager can fail to delete local logs if log aggregation fails
Date Fri, 10 Apr 2015 18:24:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14490060#comment-14490060

Jason Lowe commented on YARN-3476:

bq. As a user, I think if logs did not aggregated then user expect it should be available
in local disk for reference.

We could leave the logs on the local disk, but then we need some kind of retention logic to
handle that case.  If we don't have such logic then we risk eventually filling up the disks
(which is what happened in this case on a number of nodes).

bq. But in above scenario, does logs aggregation completed and available in HDFS?

Not all of the application's logs were available in HDFS because it encountered an error (token-related)
trying to upload the logs.

> Nodemanager can fail to delete local logs if log aggregation fails
> ------------------------------------------------------------------
>                 Key: YARN-3476
>                 URL: https://issues.apache.org/jira/browse/YARN-3476
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: log-aggregation, nodemanager
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>            Assignee: Rohith
> If log aggregation encounters an error trying to upload the file then the underlying
TFile can throw an illegalstateexception which will bubble up through the top of the thread
and prevent the application logs from being deleted.

This message was sent by Atlassian JIRA

View raw message