hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HADOOP-15240) ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths failure, FileAlreadyExistsException
Date Thu, 26 Jul 2018 18:31:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran resolved HADOOP-15240.
    Resolution: Cannot Reproduce

> ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths failure, FileAlreadyExistsException
> ------------------------------------------------------------------------------------------
>                 Key: HADOOP-15240
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15240
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 3.1.0, 3.2.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
> Transient {{ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths}} failure; {{FileAlreadyExistsException}}
during the setup phase of the test -writing out the work to
> {{org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/build/test:fork-0002/job_200707120153_0002/_temporary/1/_temporary/attempt_200707120153_0002_m_000000_0/part-m-00000}}
> There's something wrong here as that path is a FileOutputCommitter workingDir path, not
a partitioned committer. 
> Hypotheses: some filesystem caching in the test run is stopping the right committer from
being picked up.
> It's possible that this has been happening for a while, but only some cleanup failure
from a previous run has caught it.
> Proposed
> * add checks in the AbstractIT test to verify the right committer is picked up in the
task context by probing for the working dir: file:// for staging, */__magic/* for magic.
> * and verify that success file is there.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message