streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sblackmon <sblack...@apache.org>
Subject persister integration testing refactoring/improvement exemplar
Date Wed, 05 Oct 2016 16:00:07 GMT
All, 

I’ve pushed a branch that depicts the integration testing with databases in docker that
I envision adopting across persisters and examples this quarter.

https://github.com/apache/incubator-streams/compare/master...steveblackmon:STREAMS-400?expand=1

Fully resolves:

STREAMS-400
streams-persist-elasticsearch : bump to version 2.x

STREAMS-404
Integration Test for FsElasticsearchIndex

The highlight though is that making docker a standard part of the build environment, we can 
a) toss out the hodgepodge of maven plugins that startup databases for testing
b) avoid relying on integration test scaffolds maintained by database SDK libraries
c) make it much easier to ensure compatibility with the most current database versions (just
run the test suite using ‘latest’ tags)
d) reduce the delta between how the readers/writers operate during the test vs in the wild
e) other than the initial pull of the docker image these type of tests seem to run much faster
with the DB in a separate container

The only downside is that to run these integration tests devs need a docker environment,
but I don’t see that as a significant problem.

The other nice thing about this change-set is how every IT has fully sufficient and re-usable
src/test/resource .conf file rather than managing test configurations within the code itself.

There’s a few things to figure out regarding how to best tie the docker plugin up with the
maven reactor.  Ideally we want

mvn -PdockerITs verify -DskipITs=false 

to work from every level of the reactor.  In this branch it only works when run directly
from streams-persist-elasticsearch

NOTE: to run this locally you’ll need to mvn install
https://github.com/steveblackmon/incubator-streams-master/tree/STREAMS-400

Please let me know what you think.  If no one objects I’ll create tickets to apply this
pattern to mongo, kafka, neo4j, hbase across streams-project/streams-contrib/streams-persist-*
and streams-examples/*  

Steve



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message