hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15079) ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch
Date Wed, 10 Jan 2018 20:51:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321080#comment-16321080
] 

Steve Loughran commented on HADOOP-15079:
-----------------------------------------

HADOOP-15079 Patch 0001: reduce number of superfluous delete and mkdir ops in renames

* innerDelete() doesn't do the mkdirs, pulled up to delete()
* mkdirs() skips superflous delete of fake dirs
* fixed up equality check bug in rename() which was comparing parent paths on object identity,
not path, so not skipping if src and dest were in same dir unless they shared a path instance
(which in tests, they did!)
* test that too
* shiny lambda-8 stuff to make extended test more readable, include full metrics on assert
failures
* marked up some more S3A and Listing operations with retry semantics, including tune copyFile

Issue: I don't see why mkdirs() needs to S3Guard.makeDirsOrdered(metadataStore, metadataStoreDirs,
username, true); given that createFakeDirectory will do it by way of Putf -> finishedWrite
-> S3Guard.addAncestors. [~fabbri]: what do you say? 

Issue 2: The progress callback in copyFile. We could make that a singleton of the class; all
it does is update a counter. Worthwhile?

Testing: S3 ireland, without s3guard.

Anyway, yes: too many deletes. Something to track.

FWIW, I'm not sure this is all *just* a regression from the committer. The equality check
bug has probably been around a while, we just never noticed as we've never stepped through
the code line by line in a debugger trying to work out why delete was happening.

> ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-15079
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15079
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 3.1.0
>            Reporter: Sean Mackrory
>            Assignee: Steve Loughran
>            Priority: Critical
>         Attachments: HADOOP-15079-001.patch
>
>
> I see this test failing with "object_delete_requests expected:<1> but was:<2>".
I printed stack traces whenever this metric was incremented, and found the root cause to be
that innerMkdirs is now causing two calls to delete fake directories when it previously caused
only one. It is called once inside createFakeDirectory, and once directly inside innerMkdirs
later:
> {code}
>         at org.apache.hadoop.fs.s3a.S3AInstrumentation.incrementCounter(S3AInstrumentation.java:454)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1108)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$8(S3AFileSystem.java:1369)
>         at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
>         at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:279)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1366)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1625)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2634)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:2599)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1498)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$createEmptyObject$11(S3AFileSystem.java:2684)
>         at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:108)
>         at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:259)
>         at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
>         at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255)
>         at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:230)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:2682)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:2657)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2021)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1956)
>         at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2305)
>         at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
>         at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:209)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>         at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>         at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>         at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>         at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74
> {code}
> {code}
>         at org.apache.hadoop.fs.s3a.S3AInstrumentation.incrementCounter(S3AInstrumentation.java:454)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1108)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$8(S3AFileSystem.java:1369)
>         at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
>         at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:279)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1366)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1625)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2634)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2025)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1956)
>         at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2305)
>         at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
>         at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:209)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>         at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>         at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>         at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>         at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message