hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden
Date Wed, 20 Dec 2017 18:00:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298823#comment-16298823
] 

Steve Loughran commented on HADOOP-14630:
-----------------------------------------

TestLocalFSContractRename failing
{code}
common/target/test/data/2/fEJaHzwpnD/testRenameFileUnderFileSubdir/file
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:534)
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:303)
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:292)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:393)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
	at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.rename(AbstractFSContractTestBase.java:372)
	at org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContractRenameTest.java:327)
	at org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileUnderFileSubdir(AbstractContractRenameTest.java:312)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}

and 
{code}
Stacktrace
org.apache.hadoop.fs.FileAlreadyExistsException: Destination exists and is not a directory:
/testptch/hadoop/hadoop-common-project/hadoop-common/target/test/data/4/cE76lDnwXc/testRenameFileUnderFile/file
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:559)
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:534)
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:303)
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:292)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:393)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
	at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
	at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:617)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.rename(AbstractFSContractTestBase.java:372)
	at org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContractRenameTest.java:327)
	at org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileUnderFile(AbstractContractRenameTest.java:299)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}

This is amusing. The operation is calling {{srcFile.renameTo(dest)}}, which returns false.
so then falls back to a copy, which fails in the mkdir() phase.

Conclusion: trying to rename somethng under a file raises an exception. 

I think we should just document this as a special case, make it an option & have the ADL
and native ones do it, any others if they want to.

> Contract Tests to verify create, mkdirs and rename under a file is forbidden
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-14630
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14630
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/azure, fs/s3, fs/swift
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-14630-001.patch, HADOOP-14630-002.patch, HADOOP-14630-003.patch
>
>
> Object stores can get into trouble in ways which an FS would never, do, ways so obvious
we've never done tests for them. We know what the problems are: test for file and dir creation
directly/indirectly under other files
> * mkdir(file/file)
> * mkdir(file/subdir)
> * dir under file/subdir/subdir
> * dir/dir2/file, verify dir & dir2 exist
> * dir/dir2/dir3, verify dir & dir2 exist 
> * rename(src, file/dest)
> * rename(src, file/dir/dest)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message