hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Xiaoqiao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-13529) Fix default trash policy emptier trigger time correctly
Date Sun, 06 May 2018 18:29:00 GMT
He Xiaoqiao created HDFS-13529:

             Summary: Fix default trash policy emptier trigger time correctly
                 Key: HDFS-13529
                 URL: https://issues.apache.org/jira/browse/HDFS-13529
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 2.7.6, 3.1.0, 3.2.0, 2.9.2, 2.8.5
            Reporter: He Xiaoqiao

Trash emptier is designed to auto trigger at UTC 00:00, however I am confused all the time
that it usually triggers at a few minutes even half a hour after UTC 00:00 actually in our
production cluster.
The main reason is default policy emptier thread sleep more time than as expect, since it
does not consider the delete operation time cost itself. especially for a large cluster, auto
trash cleaner may cost dozens of minutes.
The right way is that gets current time {{now}} before calculate {{end}} time.
      long now = Time.now();
      while (true) {
        end = ceiling(now, emptierInterval);
        try {                                     // sleep for interval
          Thread.sleep(end - now);
        } catch (InterruptedException e) {
          break;                                  // exit on interrupt

        try {
          now = Time.now();
          ...... // delete trash checkpoint
        } catch (Exception e) {
          LOG.warn("RuntimeException during Trash.Emptier.run(): ", e); 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message