hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prabhu Joseph (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-9080) Bucket Directories as part of ATS done accumulates
Date Wed, 06 Mar 2019 08:17:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785398#comment-16785398
] 

Prabhu Joseph commented on YARN-9080:
-------------------------------------

Hi [~Rakesh_Shah], Have tested on my test cluster which had bucket directories older than
a month.

Before Patch there are clustertimestamp, bucket directories which are older than retention

yarn.timeline-service.entity-group-fs-store.retain-seconds (604800 i.e 7 days) are present.

{code}
[yarn@tparimi-tarunhdp26-3 ~]$ hadoop fs -ls -R /ats/done
drwxr-xr-x   - yarn hadoop          0 2018-10-24 15:03 /ats/done/1540393151330
drwxr-xr-x   - yarn hadoop          0 2018-10-24 15:03 /ats/done/1540393151330/0000
drwxr-xr-x   - yarn hadoop          0 2018-10-31 15:15 /ats/done/1540393151330/0000/000
drwxr-xr-x   - yarn hadoop          0 2018-10-26 09:41 /ats/done/1540478614379
drwxr-xr-x   - yarn hadoop          0 2018-10-26 09:41 /ats/done/1540478614379/0000
drwxr-xr-x   - yarn hadoop          0 2018-11-02 11:02 /ats/done/1540478614379/0000/000
drwxr-xr-x   - yarn      hadoop          0 2019-03-05 04:44 /ats/done/1551701066597
drwxr-xr-x   - yarn      hadoop          0 2019-03-05 04:44 /ats/done/1551701066597/0000
drwxr-xr-x   - yarn      hadoop          0 2019-03-05 04:46 /ats/done/1551701066597/0000/000
drwxrwx---   - ambari-qa hadoop          0 2019-03-05 04:44 /ats/done/1551701066597/0000/000/application_1551701066597_0001
{code}

After Patch with cleaner interval set to minute.
yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds = 60 

After restart of ATS and a minute, the older clustertimestamp, bucket directories are removed
as expected. 

{code}
[yarn@tparimi-tarunhdp26-3 yarn]$ hadoop fs -ls -R /ats/done
drwxr-xr-x   - yarn      hadoop          0 2019-03-05 04:44 /ats/done/1551701066597
drwxr-xr-x   - yarn      hadoop          0 2019-03-05 04:44 /ats/done/1551701066597/0000
drwxr-xr-x   - yarn      hadoop          0 2019-03-05 04:46 /ats/done/1551701066597/0000/000
drwxrwx---   - ambari-qa hadoop          0 2019-03-05 04:44 /ats/done/1551701066597/0000/000/application_1551701066597_0001
{code}


[~rohithsharma] Can you review the v4 patch for this jira.




> Bucket Directories as part of ATS done accumulates
> --------------------------------------------------
>
>                 Key: YARN-9080
>                 URL: https://issues.apache.org/jira/browse/YARN-9080
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Prabhu Joseph
>            Assignee: Prabhu Joseph
>            Priority: Major
>         Attachments: 0001-YARN-9080.patch, 0002-YARN-9080.patch, 0003-YARN-9080.patch,
YARN-9080-004.patch
>
>
> Have observed older bucket directories cluster_timestamp, bucket1 and bucket2 as part
of ATS done accumulates. The cleanLogs part of EntityLogCleaner removes only the app directories
and not the bucket directories.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message