hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bibin A Chundatt (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-9148) AggregatedLogDeletion doesnt work with S3
Date Mon, 14 Jan 2019 11:49:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741974#comment-16741974

Bibin A Chundatt commented on YARN-9148:

Thank you [~stevel@apache.org] for review
{quote}There aren't any explicit tests for this. I will leave it for the YARN team to decide
how important that is. It seems to me that if the various scan operations could be isolated
then this would be possible, with the test setup creating directories with the different characteristics.
With WIP patch wanted to check the solution was acceptable. Comments and testcases will handle
in next patch.
{quote}One thing to consider with this design is that for a "real" FS, time to scan the dir
will become potentially O(files) over O(1)
{{CachedApplicationLogCleanerTask}} is alternative solution for log deletion incase of S3.
 For real FS we should use {{DefaultApplicationLogCleanerTask}}. Will add comments for the


{{DefaultApplicationLogCleanerTask}} still works based on old application Directory modification
Time. O(1)

Incase of S3 any better solution to reduce the scanning time.?


> AggregatedLogDeletion doesnt work with S3
> -----------------------------------------
>                 Key: YARN-9148
>                 URL: https://issues.apache.org/jira/browse/YARN-9148
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Bibin A Chundatt
>            Priority: Critical
>         Attachments: YARN-9148.001.patch
> Aggregated Log deletion works based on modification time of application directory
> S3AFileStatus give current time in case of directory.
> {code}
>         if (appDir.isDirectory() &&
>             appDir.getModificationTime() < cutoffMillis) {
>           ApplicationId appId = ApplicationId.fromString(
> {code}
> S3AFileStatus#getModificationTime
> {code}
>   @Override
>   public long getModificationTime(){
>     if(isDirectory()){
>       return System.currentTimeMillis();
>     } else {
>       return super.getModificationTime();
>     }
>   }
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message