hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
Date Fri, 03 Aug 2018 20:55:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568746#comment-16568746
] 

Steve Loughran commented on HADOOP-15576:
-----------------------------------------

legit regression; the rejection of 0-part MPUs is breaking a mock test. 
{code}
org.apache.hadoop.fs.s3a.commit.PathCommitException: `null': Failed to commit upload against
output/path/dateint=20161116/hour=14/836122ed-53e4-43a9-a3c0-beeff1e154ce.parquet, described
in null: java.lang.IllegalArgumentException: No upload parts: No upload parts
	at org.apache.hadoop.fs.s3a.commit.CommitOperations.commit(CommitOperations.java:165)
	at org.apache.hadoop.fs.s3a.commit.CommitOperations.commitOrFail(CommitOperations.java:134)
	at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.lambda$commitPendingUploads$3(AbstractS3ACommitter.java:451)
	at org.apache.hadoop.fs.s3a.commit.Tasks$Builder$1.run(Tasks.java:254)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: No upload parts
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
	at org.apache.hadoop.fs.s3a.WriteOperationHelper.finalizeMultipartUpload(WriteOperationHelper.java:222)
	at org.apache.hadoop.fs.s3a.WriteOperationHelper.completeMPUwithRetries(WriteOperationHelper.java:269)
	at org.apache.hadoop.fs.s3a.commit.CommitOperations.innerCommit(CommitOperations.java:179)
	at org.apache.hadoop.fs.s3a.commit.CommitOperations.commit(CommitOperations.java:151)
	... 8 more
{code}

> S3A  Multipart Uploader to work with S3Guard and encryption
> -----------------------------------------------------------
>
>                 Key: HADOOP-15576
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15576
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2
>            Reporter: Steve Loughran
>            Assignee: Ewan Higgs
>            Priority: Blocker
>         Attachments: HADOOP-15576-005.patch, HADOOP-15576-007.patch, HADOOP-15576.001.patch,
HADOOP-15576.002.patch, HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with the tests
to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper; adding any
new methods are needed there.
> # Tests. the tests of HDFS-13713. 
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and even if there
was, it might not show that S3Guard was bypassed, because there's no checks that listFiles/listStatus
shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader, encryption settings
are't picked up. Files being uploaded this way *are not being encrypted*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message