Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 17 Nov 2017 08:35:00 +0000 (UTC)
From: "Aaron Fabbri (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13115365.1509545275000.264663.1510907700541@Atlassian.JIRA>
In-Reply-To: <JIRA.13115365.1509545275000@Atlassian.JIRA>
References: <JIRA.13115365.1509545275000@Atlassian.JIRA> <JIRA.13115365.1509545275647@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk:
 Yetus patch checker
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 17 Nov 2017 08:35:05 -0000


     [ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Fabbri updated HADOOP-15003:
----------------------------------
    Attachment: HADOOP-15033-testfix-1.diff

(Attaching a small patch.)

Busy day today but I think I have a fix for the parallel test failure.  What I did:

- Instrumented test class configuration constructors/getters.
- Dumped a stack trace on {{S3AFileSystem#initialize()}}.
- Added the isMagicCommitEnabled bit to {{S3AFileSystem#toString()}} (included in patch).
- Reproduced the failure.

What I saw:
- Beginning part of huge scale test, where the large file is created, was using some existing {{S3AFileSystem}} instance with {{isMagicCommitEnabled}} false, despite us doing the work to set up a {{Configuration}} with the option set to true.
- Second part of huge test (post create assertions) that was failing on asserting that the .pending file was there *was* getting a new S3AFileSystem with the magic commit flag enabled, but it didn't matter since the creation did not have it enabled.

Seems like the initial FS is coming from the FileSystem cache so I disabled that (patch included) and re-ran and my reproduction vanished.  Logging shows that both test phases (create, post assert) are getting a fresh FileSystem with the magic commit flag enabled now.

> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>
>                 Key: HADOOP-15003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15003
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch, HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch, HADOOP-13786-049.patch, HADOOP-13786-050.patch, HADOOP-15033-testfix-1.diff
>
>
> This is a Yetus only JIRA created to have Yetus review the HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR [https://github.com/apache/hadoop/pull/282] is stopping this happening in HADOOP-14971.
> Reviews should go into the PR/other task


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org