hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
Date Fri, 28 Jul 2017 23:00:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105830#comment-16105830
] 

Yongjun Zhang commented on HDFS-12217:
--------------------------------------

Hi [~manojg],

Thanks for reporting the issue and working on.

It looks good in general, some comments here:

1. we can use package wide scope for the isFileDeleted method instead of protected

2. f can't be null in the code, I mean, either an exception will be thrown, or f will valid
value. So we don't need to check whether it's null here.
{code}
   for (Future<List<INodesInPath>> f : futureList) {
      try {
        if (f != null) {
          iipSet.addAll(f.get());
        }
      } catch (Exception e) {
        LOG.warn("INode filter task encountered exception: ", e);
      }
    }
...
{code}

3. If any exception is thrown in the above code, we can log a waning, and rethrow the exception,
because the exception likely indicate a real problem that we don't expect to happen. If we
continue, we might get inconsistent snapshot.

Thanks.



> HDFS snapshots doesn't capture all open files when one of the open files is deleted
> -----------------------------------------------------------------------------------
>
>                 Key: HDFS-12217
>                 URL: https://issues.apache.org/jira/browse/HDFS-12217
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: snapshots
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Manoj Govindassamy
>            Assignee: Manoj Govindassamy
>         Attachments: HDFS-12217.01.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the open files.
Just like all other files, these open files in the snapshots will remain immutable. But, sometimes
it is found that snapshots fail to capture all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode corresponding to
an active lease 
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having active lease,
and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few leases there
by causing the caller to trip over and not return all the open files needed by the snapshot
manager.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message