hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14971) Merge S3A committers into trunk
Date Mon, 30 Oct 2017 16:42:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225277#comment-16225277

Steve Loughran commented on HADOOP-14971:

Had a conf call w/ Aaron on friday. He will have some fuller comments to make; I've got some
(from memory) notes.

* Proposed: restore different config options for s3a retry policy and aws SDK policy, so that
if s3 wraps AWS it doesn't double the retries. Issue: we aren't 100% sure when AWS sdk retries,
though we know that it does for PUT of files, and (in the past) does not for the final POST
of an MPU. Action: restore the options.
* Some discussion about how to react to NPEs &c in the AWS sdk. For now, do not attempt
to retry.
* What should we do about delete and idempotency? This is a long standing topic of past HDFS
discussion (HDFS-4872). Aaron: what did we conclude here?
* Sometimes DDB throws resource not found, esp. on test setup after a new table is created.
We way want to treat the DDB exception as retriable, even though S3 404 -> FNFE -> unrecoverable.
(steve says: good idea, let's do it after the merge)
* Security: we should document the security implications for the committers & have it
reviewed. Things to worry about: Denial of service, destroying/tampering with data by altering
.pendingset files; aborting pending uploads. The Staging committers use the shared HDFS FS,
so need to have security policy there reviewed and documented. I'm reviewing how we use &
document this path so that it defaults to being something under the user's home directory.
That way: if HDFS is locked down, the path should be too

> Merge S3A committers into trunk
> -------------------------------
>                 Key: HADOOP-14971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14971
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
> Merge the HADOOP-13786 committer into trunk. This branch is being set up as a github
PR for review there & to keep it out the mailboxes of the watchers on the main JIRA

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message