hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14527) ITestS3GuardListConsistency is too slow
Date Thu, 15 Jun 2017 20:55:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051053#comment-16051053
] 

Aaron Fabbri commented on HADOOP-14527:
---------------------------------------

Added a method to clear any delayed keys from the inconsistent client, instead of doing Thread.sleep()
to wait for timeout. This only gave small speedup.  Pasting runtimes of main test cases here:

{noformat}
2017-06-15 11:29:24,976 [JUnit-testConsistentRenameAfterDelete] INFO  s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentRenameAfterDelete(244)) - took 17939 msec
2017-06-15 11:29:43,360 [JUnit-testConsistentListAfterDelete] INFO  s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListAfterDelete(194)) - took 16404 msec
2017-06-15 11:33:15,135 [JUnit-testConsistentListFiles] INFO  s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testConsistentListFiles(373))
- took 209143 msec
2017-06-15 11:34:57,714 [JUnit-testConsistentListLocatedStatusAfterPut] INFO  s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListLocatedStatusAfterPut(305)) - took 73391
msec
2017-06-15 11:35:13,635 [JUnit-testConsistentListAfterRename] INFO  s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListAfterRename(121)) - took 9961 msec
2017-06-15 11:35:18,434 [JUnit-testConsistentListStatusAfterPut] INFO  s3a.AbstractS3ATestBase
(ITestS3GuardListConsistency.java:testConsistentListStatusAfterPut(279)) - took 3212 msec
2017-06-15 11:35:56,313 [JUnit-testRollingRenames] INFO  s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testRollingRenames(153))
- took 35460 msec
{noformat}

Will look at reducing the big ones when I get a chance.

> ITestS3GuardListConsistency is too slow
> ---------------------------------------
>
>                 Key: HADOOP-14527
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14527
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>            Priority: Minor
>
> I'm really glad to see folks adopting the inconsistency injection stuff and adding test
cases to ITestS3GuardListConsistency.  That test class has become very slow, however, mostly
due to {{Thread.sleep()}} calls that wait for the inconsistency injection timers to expire.
> I will take a stab at speeding up this test class.  As is it takes about 8 minutes to
run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message