accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ke...@deenlo.com
Subject Re: Review Request 28214: Decouple MiniAccumuloCluster from integration tests and introduce StandaloneAccumuloCluster
Date Fri, 21 Nov 2014 23:00:12 GMT


> On Nov. 21, 2014, 3:19 a.m., kturner wrote:
> > test/src/test/java/org/apache/accumulo/test/ConditionalWriterIT.java, line 98
> > <https://reviews.apache.org/r/28214/diff/5/?file=770707#file770707line98>
> >
> >     SimpleMacIT [starts one MAC][1] per class.  It seems like AccumuloClusterIT
starts a MAC per method?   
> >     
> >     [1]:https://github.com/apache/accumulo/blob/1.6.1/test/src/test/java/org/apache/accumulo/test/functional/SimpleMacIT.java#L60
> 
> Josh Elser wrote:
>     Yes, this is a change. I made SharedMiniClusterIT to make ShellServerIT run a *lot*
faster, I didn't see any other ITs that were affected to the same extent. I couldn't come
up with a good way to implement this inside of AccumuloClusterIT so I punted on it for now.
Suggestions welcome.
> 
> kturner wrote:
>     Ok.  I saw SharedMiniClusterIT and was wondering why only ShellServerIT used it.
 Do you know what the difference in build times is w/ this change?
> 
> Josh Elser wrote:
>     Ha, significant. Up to 230s instead of 30 when running with MAC :). Will see if I
can come up with anything to address it.
> 
> kturner wrote:
>     > Ha, significant. Up to 230s instead of 30 when running with MAC :).
>     
>     Thats for an individual test.  Do you know what the total diff for `time mvn verify`
is w/ and w/o your changes?
> 
> Josh Elser wrote:
>     Oh, sorry, I thought you were only asking specifically about ConditionalWriterIT
:) (with a standalone instance, it's 25s if you were curious :P). I'll run some comparisions
against 1.6 presently, 1.6 with these changes out of the box (still using MAC) and then 1.6
with a standalone instance available tonight.
> 
> Josh Elser wrote:
>     Oddly, NamespacesIT has also been really problematic with ZK starting. My guess is
that it this change is also affecting things for that test. Definitely something I need to
look into until we resolve the ZK issue.

> with a standalone instance, it's 25s if you were curious

Do you think we could start a Mini instance preintegration test and use that as a "standalone
instance", if no standalone instance is configured?


- kturner


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28214/#review62514
-----------------------------------------------------------


On Nov. 21, 2014, 8:40 p.m., Josh Elser wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28214/
> -----------------------------------------------------------
> 
> (Updated Nov. 21, 2014, 8:40 p.m.)
> 
> 
> Review request for accumulo.
> 
> 
> Bugs: ACCUMULO-3167
>     https://issues.apache.org/jira/browse/ACCUMULO-3167
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> We have a large number of good tests which are only capable of being run against a MiniAccumuloCluster.
This is undesirable because it limits our tests to the scope of what MiniAccumuloCluster provides,
or implements as a standalone cluster does. An accurate test environment would be a true deployment
with distributed HDFS and ZooKeper instances that back a distributed Accumulo instance. This
patch introduces a StandaloneAccumuloCluster, in addition to a few other new interfaces (ClusterControl)
which help support the necessary functionality. The StandaloneAccumuloCluster is the "MiniAccumuloCluster"
counterpart to a distributed cluster.
> 
> Given the StandaloneAccumuloCluster, many of the integration tests need to be rewritten
in such a way that support both a MiniAccumuloCluster and the Standalone cluster. While being
a painful set of changes, this does help generalize some of the tests and conform them to
some best practices to simplify things.
> 
> I also nuked some initial interfaces which I initially stubbed out because they turned
out not being useful.
> 
> 
> Diffs
> -----
> 
>   minicluster/src/main/java/org/apache/accumulo/cluster/AccumuloCluster.java c982de0

>   minicluster/src/main/java/org/apache/accumulo/cluster/AccumuloClusters.java 50cb9db

>   minicluster/src/main/java/org/apache/accumulo/cluster/AccumuloConfig.java 0df2348 
>   minicluster/src/main/java/org/apache/accumulo/cluster/ClusterControl.java PRE-CREATION

>   minicluster/src/main/java/org/apache/accumulo/cluster/RemoteShell.java PRE-CREATION

>   minicluster/src/main/java/org/apache/accumulo/cluster/RemoteShellOptions.java PRE-CREATION

>   minicluster/src/main/java/org/apache/accumulo/cluster/package-info.java f1b649d 
>   minicluster/src/main/java/org/apache/accumulo/cluster/standalone/StandaloneAccumuloCluster.java
PRE-CREATION 
>   minicluster/src/main/java/org/apache/accumulo/cluster/standalone/StandaloneClusterControl.java
PRE-CREATION 
>   minicluster/src/main/java/org/apache/accumulo/minicluster/MiniAccumuloCluster.java
3e8c5a0 
>   minicluster/src/main/java/org/apache/accumulo/minicluster/ServerType.java 3590a20 
>   minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterControl.java
PRE-CREATION 
>   minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImpl.java
7283c19 
>   minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloConfigImpl.java
2d7103e 
>   minicluster/src/test/java/org/apache/accumulo/cluster/AccumuloClustersTest.java e368240

>   minicluster/src/test/java/org/apache/accumulo/minicluster/MiniAccumuloClusterStartStopTest.java
2031b11 
>   minicluster/src/test/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImplTest.java
b19d289 
>   server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java 55548e3

>   test/src/main/java/org/apache/accumulo/test/TestMultiTableIngest.java 16f0b3f 
>   test/src/test/java/org/apache/accumulo/harness/AccumuloClusterIT.java PRE-CREATION

>   test/src/test/java/org/apache/accumulo/harness/AccumuloIT.java PRE-CREATION 
>   test/src/test/java/org/apache/accumulo/harness/MiniClusterConfigurationCallback.java
PRE-CREATION 
>   test/src/test/java/org/apache/accumulo/harness/MiniClusterHarness.java PRE-CREATION

>   test/src/test/java/org/apache/accumulo/harness/SharedMiniClusterIT.java PRE-CREATION

>   test/src/test/java/org/apache/accumulo/harness/conf/AccumuloClusterConfiguration.java
PRE-CREATION 
>   test/src/test/java/org/apache/accumulo/harness/conf/AccumuloClusterPropertyConfiguration.java
PRE-CREATION 
>   test/src/test/java/org/apache/accumulo/harness/conf/AccumuloMiniClusterConfiguration.java
PRE-CREATION 
>   test/src/test/java/org/apache/accumulo/harness/conf/AccumuloStandaloneClusterConfiguration.java
PRE-CREATION 
>   test/src/test/java/org/apache/accumulo/test/Accumulo3010IT.java 791b1d5 
>   test/src/test/java/org/apache/accumulo/test/Accumulo3030IT.java 3512e4a 
>   test/src/test/java/org/apache/accumulo/test/Accumulo3047IT.java 70e1c30 
>   test/src/test/java/org/apache/accumulo/test/AuditMessageIT.java 49b5d70 
>   test/src/test/java/org/apache/accumulo/test/BatchWriterIT.java ca72e7a 
>   test/src/test/java/org/apache/accumulo/test/CleanWalIT.java d0bfe3f 
>   test/src/test/java/org/apache/accumulo/test/ConditionalWriterIT.java 8406570 
>   test/src/test/java/org/apache/accumulo/test/ConfigurableMajorCompactionIT.java 899b41b

>   test/src/test/java/org/apache/accumulo/test/DeleteRowsIT.java ff67e89 
>   test/src/test/java/org/apache/accumulo/test/ExistingMacIT.java ec72281 
>   test/src/test/java/org/apache/accumulo/test/ImportExportIT.java a48ed9d 
>   test/src/test/java/org/apache/accumulo/test/InterruptibleScannersIT.java PRE-CREATION

>   test/src/test/java/org/apache/accumulo/test/KeyValueEqualityTest.java 1302b23 
>   test/src/test/java/org/apache/accumulo/test/MetaConstraintRetryIT.java b3c3640 
>   test/src/test/java/org/apache/accumulo/test/MetaSplitIT.java 50a1446 
>   test/src/test/java/org/apache/accumulo/test/MultiTableBatchWriterIT.java 484c048 
>   test/src/test/java/org/apache/accumulo/test/NamespacesIT.java 8188deb 
>   test/src/test/java/org/apache/accumulo/test/NoMutationRecoveryIT.java 87ad1a3 
>   test/src/test/java/org/apache/accumulo/test/ShellServerIT.java 4457e70 
>   test/src/test/java/org/apache/accumulo/test/SplitRecoveryIT.java 96d3a1a 
>   test/src/test/java/org/apache/accumulo/test/TableConfigurationUpdateIT.java 0d9a211

>   test/src/test/java/org/apache/accumulo/test/TableOperationsIT.java bb12279 
>   test/src/test/java/org/apache/accumulo/test/VolumeIT.java d5c940d 
>   test/src/test/java/org/apache/accumulo/test/functional/AbstractMacIT.java 22e46ff 
>   test/src/test/java/org/apache/accumulo/test/functional/AccumuloInputFormatIT.java ad84960

>   test/src/test/java/org/apache/accumulo/test/functional/AddSplitIT.java 05de342 
>   test/src/test/java/org/apache/accumulo/test/functional/BadIteratorMincIT.java 9c4492e

>   test/src/test/java/org/apache/accumulo/test/functional/BalanceInPresenceOfOfflineTableIT.java
887aee4 
>   test/src/test/java/org/apache/accumulo/test/functional/BatchScanSplitIT.java 688a326

>   test/src/test/java/org/apache/accumulo/test/functional/BatchWriterFlushIT.java 465936e

>   test/src/test/java/org/apache/accumulo/test/functional/BigRootTabletIT.java b021c3a

>   test/src/test/java/org/apache/accumulo/test/functional/BinaryIT.java 9c0edaa 
>   test/src/test/java/org/apache/accumulo/test/functional/BinaryStressIT.java a60c2d5

>   test/src/test/java/org/apache/accumulo/test/functional/BloomFilterIT.java 8f6b830 
>   test/src/test/java/org/apache/accumulo/test/functional/BulkIT.java 831dcd4 
>   test/src/test/java/org/apache/accumulo/test/functional/ChaoticBalancerIT.java 8afb3d2

>   test/src/test/java/org/apache/accumulo/test/functional/ClassLoaderIT.java d71819e 
>   test/src/test/java/org/apache/accumulo/test/functional/CleanUpIT.java 79bbb90 
>   test/src/test/java/org/apache/accumulo/test/functional/CloneTestIT.java 505dd5a 
>   test/src/test/java/org/apache/accumulo/test/functional/CombinerIT.java 69f9134 
>   test/src/test/java/org/apache/accumulo/test/functional/ConcurrencyIT.java 92bd714 
>   test/src/test/java/org/apache/accumulo/test/functional/ConfigurableMacIT.java 9185e1b

>   test/src/test/java/org/apache/accumulo/test/functional/ConstraintIT.java 7e5944e 
>   test/src/test/java/org/apache/accumulo/test/functional/CreateAndUseIT.java 5b5249b

>   test/src/test/java/org/apache/accumulo/test/functional/CreateManyScannersIT.java aed38e8

>   test/src/test/java/org/apache/accumulo/test/functional/CredentialsIT.java 8e2e1e0 
>   test/src/test/java/org/apache/accumulo/test/functional/DeleteEverythingIT.java 0578ef4

>   test/src/test/java/org/apache/accumulo/test/functional/DeleteIT.java 3510fbd 
>   test/src/test/java/org/apache/accumulo/test/functional/DeleteRowsIT.java 4b7d664 
>   test/src/test/java/org/apache/accumulo/test/functional/DeleteRowsSplitIT.java d35ba9f

>   test/src/test/java/org/apache/accumulo/test/functional/DeleteTableDuringSplitIT.java
a0bff64 
>   test/src/test/java/org/apache/accumulo/test/functional/DynamicThreadPoolsIT.java 87497b9

>   test/src/test/java/org/apache/accumulo/test/functional/FateStarvationIT.java 4d75a16

>   test/src/test/java/org/apache/accumulo/test/functional/LargeRowIT.java d77d060 
>   test/src/test/java/org/apache/accumulo/test/functional/LogicalTimeIT.java 6aec7cd 
>   test/src/test/java/org/apache/accumulo/test/functional/MasterAssignmentIT.java 46f6b23

>   test/src/test/java/org/apache/accumulo/test/functional/MasterFailoverIT.java 218d65e

>   test/src/test/java/org/apache/accumulo/test/functional/MaxOpenIT.java 2649890 
>   test/src/test/java/org/apache/accumulo/test/functional/MergeIT.java c264dfe 
>   test/src/test/java/org/apache/accumulo/test/functional/MetadataIT.java bd0282d 
>   test/src/test/java/org/apache/accumulo/test/functional/MetadataMaxFiles.java 6b8d9b3

>   test/src/test/java/org/apache/accumulo/test/functional/MetadataMaxFilesIT.java PRE-CREATION

>   test/src/test/java/org/apache/accumulo/test/functional/PermissionsIT.java d51dcbb 
>   test/src/test/java/org/apache/accumulo/test/functional/ReadWriteIT.java bc9ab4a 
>   test/src/test/java/org/apache/accumulo/test/functional/RecoveryWithEmptyRFileIT.java
814dd85 
>   test/src/test/java/org/apache/accumulo/test/functional/RenameIT.java 8cbe84f 
>   test/src/test/java/org/apache/accumulo/test/functional/RestartIT.java e4b9c5a 
>   test/src/test/java/org/apache/accumulo/test/functional/RestartStressIT.java d8c2804

>   test/src/test/java/org/apache/accumulo/test/functional/RowDeleteIT.java 4dbd912 
>   test/src/test/java/org/apache/accumulo/test/functional/ScanIteratorIT.java 189a55c

>   test/src/test/java/org/apache/accumulo/test/functional/ScanRangeIT.java 90b881c 
>   test/src/test/java/org/apache/accumulo/test/functional/ScanSessionTimeOutIT.java 3547b68

>   test/src/test/java/org/apache/accumulo/test/functional/ScannerIT.java cbd1290 
>   test/src/test/java/org/apache/accumulo/test/functional/ServerSideErrorIT.java d765b16

>   test/src/test/java/org/apache/accumulo/test/functional/SimpleMacIT.java a4e6647 
>   test/src/test/java/org/apache/accumulo/test/functional/SparseColumnFamilyIT.java 0b63d01

>   test/src/test/java/org/apache/accumulo/test/functional/SplitIT.java 6203523 
>   test/src/test/java/org/apache/accumulo/test/functional/SslIT.java a14795c 
>   test/src/test/java/org/apache/accumulo/test/functional/StartIT.java 82278af 
>   test/src/test/java/org/apache/accumulo/test/functional/TableIT.java 832ec60 
>   test/src/test/java/org/apache/accumulo/test/functional/TabletIT.java fccc79f 
>   test/src/test/java/org/apache/accumulo/test/functional/TimeoutIT.java 4dc72e0 
>   test/src/test/java/org/apache/accumulo/test/functional/VisibilityIT.java f2460cf 
>   test/src/test/java/org/apache/accumulo/test/functional/WriteAheadLogIT.java af6eca5

>   test/src/test/java/org/apache/accumulo/test/functional/WriteLotsIT.java 214fc2f 
> 
> Diff: https://reviews.apache.org/r/28214/diff/
> 
> 
> Testing
> -------
> 
> Haven't had a 100% IT pass rate yet (last run was about 95% pass rate), but I wanted
to get the code up and have some eyes on it sooner than later.
> 
> If you want to try it out, you configure it using properties on the maven command line.
For example,  -Daccumulo.it.cluster.type=STANDALONE -Daccumulo.it.cluster.standalone.principal=root
-Daccumulo.it.cluster.standalone.password=password -Daccumulo.it.cluster.standalone.instance.name=accumulo
-Daccumulo.it.cluster.standalone.zookeepers=localhost. This would use a "real" instance with
the expected connection information. Alternatively, you can make a properties file on the
local filesystem and specify that file directly using -Daccumulo.it.properties=/path/to/file.properties.
Properties on the command line will override those specified in the property if you provide
the same in more than one place.
> 
> 
> Thanks,
> 
> Josh Elser
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message