hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits
Date Fri, 08 Dec 2017 15:23:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283715#comment-16283715
] 

Steve Loughran commented on HADOOP-13974:
-----------------------------------------

Looks pretty good

General
* check the import ordering w.r.t the style rules
* I can see that retry stuff is useful, but also it's going to have to be something we keep
an eye on maintenance-wise.
* need to move {{S3ATestUtils.listMultipartUploads()}} to this
* there's some shiny new Java 8 code in S3AUtils, like applyLocatedFiles(), which work on
RemoteIterator<LocatedFileStatus>. 
  These could be expanded to take any RemoteIterator/subclass thereof, maybe, which would
actually be something to have in hadoop common for broader use.
  {{MultipartTestUtils.assertNoUploadsAt()}} would be an example use
  
  
* MultipartUtils L210: can simplify to {{return batchIterator.hasNext();}}.


S3AFileSystem L772. Good point. They used to be bonded to the destination path, but we've
moved off that: we could just create a single instance here. Want to change it?


ITestS3GuardToolLocal: 

* good Q. about using eventually() in listings. I don't know what happens there, but I'd hope
that there's more list consistency here. (Fewer entries and you certainly need to be able
to map on subsequent posts to the outstanding MPU
* If uploadCommandAssertCount really is at risk of failing, the output should be logged or
included in the fail(). Simplest to log


> S3a CLI to support list/purge of pending multipart commits
> ----------------------------------------------------------
>
>                 Key: HADOOP-13974
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13974
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0-beta1
>            Reporter: Steve Loughran
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13974.001.patch, HADOOP-13974.002.patch, HADOOP-13974.003.patch,
HADOOP-13974.004.patch
>
>
> The S3A CLI will need to be able to list and delete pending multipart commits. 
> We can do the cleanup already via fs.s3a properties. The CLI will let scripts stat for
outstanding data (have a different exit code) and permit batch jobs to explicitly trigger
cleanups.
> This will become critical with the multipart committer, as there's a significantly higher
likelihood of commits remaining outstanding.
> We may also want to be able to enumerate/cancel all pending commits in the FS tree



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message