hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15576) S3A Multipart Uploader API implementation to work with S3Guard and encryption
Date Tue, 24 Jul 2018 17:16:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554542#comment-16554542
] 

Steve Loughran commented on HADOOP-15576:
-----------------------------------------

Ewan, still expecting this

* S3AMultipartUploader to always go through s3a FS and {{WriteOperationsHelper}}, not at the
s3 client; retry logic to go in there, along with s3guard introduction. that's for completing
& aborting ubploads ({{abortMultipartUpload()}},  {{newUploadPartRequest()}}{{finalizeMultipartUpload()}}.
That shoud be enough to wire up S3Guard. If something is missing: add it.
* S3AMultipartUploader to declare retry policy on its ops
* test to include inconsistent s3 client with failures (always) and, if s3guard is enabled
inconsistent, turned on. Your store doesn't do inconsistency so that's moot, but it may fail
-so we need those retries

You must not be creating any AWS request classes yourself, or calling the s3a client. If there
isn't a method in WriteOperationsHelper to create the instance, add it, make sure it works
with S3Guard, sets encryption stuff etc. The requirement to declare the retry policy forces
you to look at the retry logic of the layers underneath, make sure either they all retry *or
the new code adds it*. It will also help future maintenance



> S3A  Multipart Uploader API implementation to work with S3Guard and encryption
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-15576
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15576
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2
>            Reporter: Steve Loughran
>            Assignee: Ewan Higgs
>            Priority: Blocker
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with the tests
to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper; adding any
new methods are needed there.
> # Tests. the tests of HDFS-13713. 
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and even if there
was, it might not show that S3Guard was bypassed, because there's no checks that listFiles/listStatus
shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader, encryption settings
are't picked up. Files being uploaded this way *are not being encrypted*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message