hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
Date Fri, 27 Jan 2017 18:46:25 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-13786:
    Attachment: HADOOP-13786-HADOOP-13345-002.patch

Patch 002

This patch
# defines the mechanism for using multipart uploads for commits
# implements it
# adds tests
# has tests working, including a scale one.

It has not wired this up to MRv1/v2 output committers & FileOutputFormat, though the changes
have been made to the MRv2 code to make it possible to place the S3A committer in behind FileOutputFormat.

What works, then? Dela

# create a file with a path {{/example/__pending/job1/task1/part-001.bin}}
# this will initiate an MPU to {{/example/part001.bin}}, to which all the output goes.
# when the output stream is closed, the file {{/example/__pending/job1/task1/part-001.bin.pending}}
is created, which saves everything needed to commit the job
# later a {{FileCommitActions}} class can be created. 
# {{FileCommitActions .commitAllPendingFilesInPath()}} will scan a dir for .pending entries,
and commit them one by one, here {{commitAllPendingFilesInPath("/example/__pending/job1/task1/")}}
# ..which causes the file  {{/example/part001.bin}} to come into existence.
# or, if you call {{abortAllPendingFilesInPath(...)}} the MPUs are read and aborted.

Performance? < 1s to commit a single 128MB file over a long-haul link.
Duration of time to commit s3a://hwdev-steve-frankfurt-new/tests3ascale/scale/commit/__pending/job_001/commit.bin.pending:
688,701,514 nS

I think that's pretty good :)

> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
> ------------------------------------------------------------------------
>                 Key: HADOOP-13786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13786
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch
> A goal of this code is "support O(1) commits to S3 repositories in the presence of failures".
Implement it, including whatever is needed to demonstrate the correctness of the algorithm.
(that is, assuming that s3guard provides a consistent view of the presence/absence of blobs,
show that we can commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie.
not visible until the close()), if we need to use that to allow us to abort commit operations.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message