hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
Date Wed, 08 Nov 2017 15:33:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244157#comment-16244157
] 

Steve Loughran commented on HADOOP-15003:
-----------------------------------------

Think I've found one little issue which could create problems, though not by its own the root
cause. Seem closer to replicating it though, and yes, seems related to cleanup of jobs.

One little feature is that the committers were being (over?) zealous in aborting all MPUs
under their destination path, on the basis that failed tasks could have left outstanding MPUs
which, if the data were not persisted, would not be enumerable by looking for .pendingset
files.

But {{S3aFileSystem.listMultipartUploads(prefix)}} actually turns out to list everything matching
the prefix, even in parallel dirs, as it isn't adding a "/" suffix to say "directory only".
Which meant if you had a job commit to "test/myjob", i'd also delete pending uploads to "test/myjob2"

There's more to it than that; I think I'll need to review all listing stuff to be sure, but
now I have some tests failing. Provided the tests themselves are correct, then I'll be able
to find/fix it.



> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>
>                 Key: HADOOP-15003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15003
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, HADOOP-13786-043.patch,
HADOOP-13786-044.patch, HADOOP-13786-045.patch, HADOOP-13786-046.patch
>
>
> This is a Yetus only JIRA created to have Yetus review the HADOOP-13786/HADOOP-14971
patch as a .patch file, as the review PR [https://github.com/apache/hadoop/pull/282] is stopping
this happening in HADOOP-14971.
> Reviews should go into the PR/other task



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message