incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <crypt...@gmail.com>
Subject Re: Cassandra unit testing becoming nearly impossible: suggesting alternative.
Date Wed, 25 Dec 2013 20:42:29 GMT
I have been using vagrant (e.g. https://github.com/stealthly/scala-cassandra/ ) which is 100%
reproducible across devs and test systems (prod in some cases).  Also have a Docker setup
too https://github.com/pegasussolutions/docker-cassandra .  I have been doing this more and
more with clients to better mimic production before production and smoothing the release process
from development.  I also use packer (scripts released soon) to build images too (http://packer.io)

Love vagrant, packer and docker!!!  Apache Mesos too :)


/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
********************************************/


On Dec 25, 2013, at 3:28 PM, horschi <horschi@gmail.com> wrote:

> Hi Ed,
> 
> my opinion on unit testing with C* is: Use the real database, not any embedded crap :-)
> 
> All you need are fast truncates, by which I mean: 
> JVM_OPTS="$JVM_OPTS -Dcassandra.unsafesystem=true" 
> and
> auto_snapshot: false
> 
> This setup works really nice for me (C* 1.1 and 1.2, have not tested 2.0 yet).
> 
> Imho this setup is better for multiple reasons:
> - No extra classpath issues
> - Faster: Running JUnits and C* in one JVM would require a really large heap (for me
at least).
> - Faster: No Cassandra startup everytime I run my tests.
> 
> The only downside is that developers must change the properties in their configs.
> 
> cheers,
> Christian
> 
> 
> 
> On Tue, Dec 24, 2013 at 9:31 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
> I am not sure there how many people have been around developing Cassandra for as long
as I have, but the state of all the client libraries and the cassandra server is WORD_I_DONT_WANT_TO_SAY.
> 
> Here is an example of something I am seeing:
> ERROR 14:59:45,845 Exception in thread Thread[Thrift:5,5,main]
> java.lang.AbstractMethodError: org.apache.thrift.ProcessFunction.isOneway()Z
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:51)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> DEBUG 14:59:51,654 retryPolicy for schema_triggers is 0.99
> 
> In short: If you are new to cassandra and only using the newest client I am sure everything
is peachy for you.
> 
> For people that have been using Cassandra for a while it is harder to "jump ship" when
something better comes along. You need sometimes to support both hector and astyanax, it happens.

> 
> For a while I have been using hector. Even not to use hector as an API, but the one nice
thing I got from hector was a simple EmbeddedServer that would clean up after itself. Hector
seems badly broken at the moment. I have no idea how the current versions track with anything
out there in the cassandra world. 
> 
> For a while I played with https://github.com/Netflix/astyanax, which has it's own version
and schemes and dependent libraries. (astyanax has some packaging error that forces me into
maven3)
> 
> Enter cassandra 2.0 which forces you into java 0.7. Besides that it has it's own kit
of things it seems to want. 
> 
> I am guessing since hectors embedded server does not work, and I should go to https://github.com/jsevellec/cassandra-unit
not sure...really...how anyone does this anymore. I am sure I could dive into the source code
and figure this out, but I would just rather have a stable piece of code that brings up the
embedded server that "just works" and "continues working".
> 
> I can not seem to get this working right either. (since it includes hector I see from
the pom)
> 
> Between thrift, cassandra,client x, it is almost impossible to build a sane classpath,
and that is not even counting the fact that people have their own classpath issues (with guava
mismatches etc).
> 
> I think the only sane thing to do is start shipping cassandra-embedded like this:
> 
> https://github.com/kstyrc/embedded-redis
> 
> In other words package embedded-cassandra as a binary. Don't force the client/application
developer to bring cassandra on the classpath and fight with mismatches in thrift/guava etc.
That or provide a completely shaded cassandra server for embedded testing. As it stands now
trying to support a setup that uses more than one client or works with multiple versions of
cassandra is major pita.  (aka library x compiled against 1.2.0 library y compiled against
2.0.3)
> 
> Does anyone have any thoughts on this, or tried something similar?  
> 
> Edward
> 
> 

Mime
View raw message