hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tarun Parimi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8279) AggregationLogDeletionService does not honor yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix
Date Thu, 24 May 2018 14:11:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489074#comment-16489074
] 

Tarun Parimi commented on YARN-8279:
------------------------------------

[~jlowe] YARN-7072 seems to have been backported in HDP distribution with hadoop version 2.7.3.
That's why the discrepancy is there in versions I guess.

> AggregationLogDeletionService does not honor yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-8279
>                 URL: https://issues.apache.org/jira/browse/YARN-8279
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: log-aggregation
>    Affects Versions: 2.9.1
>            Reporter: Prabhu Joseph
>            Assignee: Tarun Parimi
>            Priority: Major
>
> AggregationLogDeletionService does not honor yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix.
AggregationLogService writes the logs into /app-logs/<user-dir>/logs-ifile where as
AggregationLogDeletion tries to delete from  /app-logs/<user-dir>/logs.
> Workaround is to set yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix and
yarn.nodemanager.remote-app-log-dir-suffix to same value "logs-ifile"
> AggregationLogDeletionService has to check the format and based upon that choose the
suffix. Currently it only checks the older suffix yarn.nodemanager.remote-app-log-dir-suffix.
> AggregatedLogDeletionService tries to delete older suffix directory.
> {code}
> 2018-05-11 08:48:19,989 ERROR logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:logIOException(182))
- Could not read the contents of hdfs://prabhucluster:8020/app-logs/hive/logs
> java.io.FileNotFoundException: File hdfs://prabhucluster:8020/app-logs/hive/logs does
not exist.
>         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:923)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:985)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:981)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:992)
>         at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.deleteOldLogDirsFrom(AggregatedLogDeletionService.java:98)
>         at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:85)
>         at java.util.TimerThread.mainLoop(Timer.java:555)
>         at java.util.TimerThread.run(Timer.java:505)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message