hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabor Bota (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14918) remove the Local Dynamo DB test option
Date Wed, 13 Jun 2018 20:04:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511603#comment-16511603
] 

Gabor Bota commented on HADOOP-14918:
-------------------------------------

In my latest (v005) I've managed to solve the destructive test issue. 
All tests pass on eu-west-1 with {{mvn -Dparallel-tests -DtestsThreadCount=8 clean verify
-Ds3guard -Ddynamo}}

* I've added a new config param {{fs.s3a.s3guard.ddb.table}} which should be set in order
to run the test in the config. This is the name of the table that will be shared and used
by the metadatastore object created in {{@BeforeClass}} and destroyed in {{@AfterClass}} and
shared between the tests. It's not documented yet in this patch, so I should add further documentation
if this solution will be accepted (markdown).
* The contents of this table will be cleared out in {{@After}} of each test, but there will
be no ms teardown until {{@AfterClass}}
* Some of the tests still need to create a new ms (so a new table), but the tables used will
be named after the name of those tests.
* Sometimes a test could take more time than what is defined in {{HadoopTestBase}} by default,
so I had to increase it in the hadoop-aws pom.xml.



> remove the Local Dynamo DB test option
> --------------------------------------
>
>                 Key: HADOOP-14918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14918
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0, 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Gabor Bota
>            Priority: Major
>         Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, HADOOP-14918-003.patch,
HADOOP-14918-004.patch, HADOOP-14918.005.patch
>
>
> I'm going to propose cutting out the localdynamo test option for s3guard
> * the local DDB JAR is unmaintained/lags the SDK We work with...eventually there'll be
differences in API.
> * as the local dynamo DB is unshaded. it complicates classpath setup for the build. Remove
it and there's no need to worry about versions of anything other than the shaded AWS
> * it complicates test runs. Now we need to test for both localdynamo *and* real dynamo
> * but we can't ignore real dynamo, because that's the one which matters
> While the local option promises to reduce test costs, really, it's just adding complexity.
If you are testing with s3guard, you need to have a real table to test against., And with
the exception of those people testing s3a against non-AWS, consistent endpoints, everyone
should be testing with S3Guard.
> -Straightforward to remove.-



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message