hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Deleted] (HADOOP-13786) Add S3Guard committer for zero-rename commits to S3 endpoints
Date Mon, 16 Oct 2017 13:11:06 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-13786:
------------------------------------
    Comment: was deleted

(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 23s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  1s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 29 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  1s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 36s{color}
| {color:green} HADOOP-13345 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  7m 19s{color} | {color:red}
root in HADOOP-13345 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m  7s{color}
| {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 27s{color} |
{color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m  7s{color}
| {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 38s{color} |
{color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 57s{color} |
{color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 18s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 44s{color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  7m 50s{color} | {color:red}
root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  7m 50s{color} | {color:red}
root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  2m  8s{color}
| {color:orange} root: The patch generated 270 new + 98 unchanged - 14 fixed = 368 total (was
112) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 38s{color} | {color:red}
hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 23s{color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  0s{color} | {color:red}
The patch has 17 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>.
Refer https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  1s{color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 56s{color} | {color:red}
hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color}
|
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 37s{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1
new + 2496 unchanged - 0 fixed = 2497 total (was 2496) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 30s{color} | {color:red}
hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 38s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 10s{color} | {color:green}
hadoop-mapreduce-client-core in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 46s{color} | {color:red}
hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 52s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 51s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
|  |  Dead store to length in org.apache.hadoop.fs.s3a.S3AFileSystem$WriteOperationHelper.newUploadPartRequest(String,
int, int, InputStream, File, Long)  At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem$WriteOperationHelper.newUploadPartRequest(String,
int, int, InputStream, File, Long)  At S3AFileSystem.java:[line 2885] |
|  |  Dead store to attemptFS in org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter.commitTask(TaskAttemptContext)
 At PartitionedStagingCommitter.java:org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter.commitTask(TaskAttemptContext)
 At PartitionedStagingCommitter.java:[line 94] |
| Failed junit tests | hadoop.security.TestKDiag |
|   | hadoop.fs.s3a.commit.staging.TestStagingMRJob |
|   | hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter |
|   | hadoop.fs.s3a.commit.staging.TestStagingPartitionedJobCommit |
|   | hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13786 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12858556/HADOOP-13786-HADOOP-13345-012.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  xml  |
| uname | Linux 4fcdad1cb685 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | HADOOP-13345 / 566c8b7 |
| Default Java | 1.8.0_121 |
| compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/branch-compile-root.txt
|
| findbugs | v3.0.0 |
| compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/patch-compile-root.txt
|
| javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/patch-compile-root.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/diff-checkstyle-root.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-aws.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/whitespace-eol.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
|
| javadoc | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws.txt
|
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/11813/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.

)

> Add S3Guard committer for zero-rename commits to S3 endpoints
> -------------------------------------------------------------
>
>                 Key: HADOOP-13786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13786
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 3.0.0-beta1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-036.patch, HADOOP-13786-037.patch, HADOOP-13786-038.patch,
HADOOP-13786-039.patch, HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch,
HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch,
HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-007.patch,
HADOOP-13786-HADOOP-13345-009.patch, HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch,
HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, HADOOP-13786-HADOOP-13345-015.patch,
HADOOP-13786-HADOOP-13345-016.patch, HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch,
HADOOP-13786-HADOOP-13345-019.patch, HADOOP-13786-HADOOP-13345-020.patch, HADOOP-13786-HADOOP-13345-021.patch,
HADOOP-13786-HADOOP-13345-022.patch, HADOOP-13786-HADOOP-13345-023.patch, HADOOP-13786-HADOOP-13345-024.patch,
HADOOP-13786-HADOOP-13345-025.patch, HADOOP-13786-HADOOP-13345-026.patch, HADOOP-13786-HADOOP-13345-027.patch,
HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-029.patch,
HADOOP-13786-HADOOP-13345-030.patch, HADOOP-13786-HADOOP-13345-031.patch, HADOOP-13786-HADOOP-13345-032.patch,
HADOOP-13786-HADOOP-13345-033.patch, HADOOP-13786-HADOOP-13345-035.patch, cloud-intergration-test-failure.log,
objectstore.pdf, s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the presence of failures".
Implement it, including whatever is needed to demonstrate the correctness of the algorithm.
(that is, assuming that s3guard provides a consistent view of the presence/absence of blobs,
show that we can commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie.
not visible until the close()), if we need to use that to allow us to abort commit operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message