hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits
Date Wed, 11 Oct 2017 21:52:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201032#comment-16201032
] 

Aaron Fabbri edited comment on HADOOP-13974 at 10/11/17 9:51 PM:
-----------------------------------------------------------------

Attaching v1 patch.

Some caveats I'd like feedback on:

- The `hadoop s3guard uploads -abort` command is dangerous, especially if there are MPU commits
in flight. I feel like this could use a failsafe "are you sure" prompt.  I'm leaning towards
implementing that when the specified age is less than one day.  Presumably this is "safer"
if your filter only matches older upload parts.  Or, we could add a "-force" option to override
it.

- Also think a disclaimer about clock synchronization might be warranted in the docs.  S3's
MPU list gives an "initiated time" which I use here.  I'm wondering if this is server side
or client side?  

- There is a test gap.. I tested the age-based filtering by hand but don't have an automated
test for it.  v2 patch should probably have at least a very basic test of the age filtering.


was (Author: fabbri):
Attaching v1 patch.

Some caveats I'd like feedback on:

- The `hadoop s3guard uploads -abort` command is dangerous, especially if there are MPU commits
in flight. I feel like this could use a failsafe "are you sure" prompt.  I'm leaning towards
implementing that when the specified age is less than one day.  Presumably this is "safer"
if your filter only matches older upload parts.

- Also think a disclaimer about clock synchronization might be warranted in the docs.  S3's
MPU list gives an "initiated time" which I use here.  I'm wondering if this is server side
or client side?  

- There is a test gap.. I tested the age-based filtering by hand but don't have an automated
test for it.  v2 patch should probably have at least a very basic test of the age filtering.

> S3a CLI to support list/purge of pending multipart commits
> ----------------------------------------------------------
>
>                 Key: HADOOP-13974
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13974
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0-beta1
>            Reporter: Steve Loughran
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13974.001.patch
>
>
> The S3A CLI will need to be able to list and delete pending multipart commits. 
> We can do the cleanup already via fs.s3a properties. The CLI will let scripts stat for
outstanding data (have a different exit code) and permit batch jobs to explicitly trigger
cleanups.
> This will become critical with the multipart committer, as there's a significantly higher
likelihood of commits remaining outstanding.
> We may also want to be able to enumerate/cancel all pending commits in the FS tree



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message