hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15297) Make S3A etag => checksum feature optional
Date Mon, 12 Mar 2018 14:24:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395306#comment-16395306

Hudson commented on HADOOP-15297:

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13812 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13812/])
HADOOP-15297. Make S3A etag => checksum feature optional. Contributed by (stevel: rev dd05871b8b57303fe0b0c652e03257b59c191802)
* (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
* (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java
* (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMiscOperations.java
* (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
* (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
* (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java
* (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

> Make S3A etag => checksum feature optional
> ------------------------------------------
>                 Key: HADOOP-15297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15297
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 3.1.0
>         Attachments: HADOOP-15297-001.patchh, HADOOP-15297-002.patch, HADOOP-15297-002.patch,
> HADOOP-15273 shows how distcp doesn't handle non-HDFS filesystems with checksums.
> Exposing Etags as checksums, HADOOP-13282, breaks workflows which back up to s3a.
> Rather than revert  I want to make it an option, off by default. Once we are happy with
distcp in future, we can turn it on.
> Why an option? Because it lines up for a successor to distcp which saves src and dest
checksums to a file and can then verify whether or not files have really changed. Currently
distcp relies on dest checksum algorithm being the same as the src for incremental updates,
but if either of the stores don't serve checksums, silently downgrades to not checking. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message