hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
Date Fri, 17 Nov 2017 08:35:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Aaron Fabbri updated HADOOP-15003:
    Attachment: HADOOP-15033-testfix-1.diff

(Attaching a small patch.)

Busy day today but I think I have a fix for the parallel test failure.  What I did:

- Instrumented test class configuration constructors/getters.
- Dumped a stack trace on {{S3AFileSystem#initialize()}}.
- Added the isMagicCommitEnabled bit to {{S3AFileSystem#toString()}} (included in patch).
- Reproduced the failure.

What I saw:
- Beginning part of huge scale test, where the large file is created, was using some existing
{{S3AFileSystem}} instance with {{isMagicCommitEnabled}} false, despite us doing the work
to set up a {{Configuration}} with the option set to true.
- Second part of huge test (post create assertions) that was failing on asserting that the
.pending file was there *was* getting a new S3AFileSystem with the magic commit flag enabled,
but it didn't matter since the creation did not have it enabled.

Seems like the initial FS is coming from the FileSystem cache so I disabled that (patch included)
and re-ran and my reproduction vanished.  Logging shows that both test phases (create, post
assert) are getting a fresh FileSystem with the magic commit flag enabled now.

> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>                 Key: HADOOP-15003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15003
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, HADOOP-13786-043.patch,
HADOOP-13786-044.patch, HADOOP-13786-045.patch, HADOOP-13786-046.patch, HADOOP-13786-047.patch,
HADOOP-13786-048.patch, HADOOP-13786-049.patch, HADOOP-13786-050.patch, HADOOP-15033-testfix-1.diff
> This is a Yetus only JIRA created to have Yetus review the HADOOP-13786/HADOOP-14971
patch as a .patch file, as the review PR [https://github.com/apache/hadoop/pull/282] is stopping
this happening in HADOOP-14971.
> Reviews should go into the PR/other task

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message