hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13760) S3Guard: add delete tracking
Date Mon, 24 Apr 2017 20:57:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981907#comment-15981907

Sean Mackrory commented on HADOOP-13760:

Thanks, [~fabbri]. I've eliminated the redundant try/catch and gotten rid of the switch statement
with fall-through. I also switched from integer constants for the checkPathForDirectory return
value to an enum. Didn't update the patch yet, though.

Regarding innerRename, there's no refactoring - all the changes are because of the change
in what object I'm looping through. The whitespace and variable name change incident to that
makes it look like a huge change, but if you interleave the changes line-by-line it looks
much less invasive. I'll see if I can clean it up at all, though.

There's a function call at the end of innerRename that cleans up any unnecessary directory
markers, which is why no tests are seeing the changes you predict. There's still a concern
if there are unnecessary round trips, though. One thing I'm considering is to change the function
back and simply add a separate call that just gets deleted items and doesn't consider them
for the rename operation. My only concern with that is that when that becomes an operation
that's too big to return all the results at once, we'll need to make a change (not sure if
we can sort by key / length) so we don't have context that's needed for an S3 object further
back in the Metadatastore iterator. Make sense? I'll think about some solutions here and look
at the problem some more.

> S3Guard: add delete tracking
> ----------------------------
>                 Key: HADOOP-13760
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13760
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Aaron Fabbri
>            Assignee: Sean Mackrory
>         Attachments: HADOOP-13760-HADOOP-13345.001.patch, HADOOP-13760-HADOOP-13345.002.patch
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add delete
> Current behavior on delete is to remove the metadata from the MetadataStore.  To make
deletes consistent, we need to add a {{isDeleted}} flag to {{PathMetadata}} and check it when
returning results from functions like {{getFileStatus()}} and {{listStatus()}}.  In HADOOP-13651,
I added TODO comments in most of the places these new conditions are needed.  The work does
not look too bad.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message