hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13163) Move invalidated blocks to replica-trash with disk layout based on timestamp
Date Tue, 13 Mar 2018 23:19:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397823#comment-16397823
] 

Arpit Agarwal commented on HDFS-13163:
--------------------------------------

Thanks for this patch [~bharatviswa]. A few comments:
# replicaTrashEnabled can be read/passed during FsDatasetAsyncDiskService construction, so
it doesn't need to be passed with every deleteAsync/deleteSync call.
# The trash directory must be created per hour, following the design doc.
# The following should be LOG.debug)
{code}
      if (LOG.isDebugEnabled()) {
        LOG.info("Moving files " + replicaToDelete.getBlockURI() + " and " +
            replicaToDelete.getMetadataURI() + " to replica-trash.");
      }
{code}
# Nitpick: coding style - there should be one space before '\{' and '\('.
{code}
      if (trashDirectory != null){
        result = moveFiles();
      } else if(replicaTrashEnabled) {
        result = moveFilesToReplicaTrash();
      } else {
        result = deleteFiles();
      }
{code}
# FsDatasetImpl.java: Reading the config key on every invalidate call seems to be rather inefficient.
Fixing #1 will also address this.
# You can call {{DataNodeTestUtils#triggerHeartbeat}} to shorten the test run time here. Also
wrap the assertion in a {{LambdaTestUtils#await}} call, so it can potentially complete sooner
and handle delays longer than the hard-coded 6 seconds.
{code}
      //Wait for few Heart beats, as the deletion is handled asynchronously
      sleep(6000);

      //After delete, replica-trash dir should not be empty
      Assert.assertTrue(replicaTrashDir.list().length > 0);
{code}

> Move invalidated blocks to replica-trash with disk layout based on timestamp
> ----------------------------------------------------------------------------
>
>                 Key: HDFS-13163
>                 URL: https://issues.apache.org/jira/browse/HDFS-13163
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Bharat Viswanadham
>            Assignee: Bharat Viswanadham
>            Priority: Major
>         Attachments: HDFS-13163-HDFS-12996.00.patch, HDFS-13163-HDFS-12996.01.patch
>
>
> When Blocks are invalidated, move the blocks to replica-trash directory and place it
in the folder when the invalidate is received from the namenode
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message