Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 988CE10DD1 for ; Sun, 29 Dec 2013 23:16:26 +0000 (UTC) Received: (qmail 29509 invoked by uid 500); 29 Dec 2013 23:16:23 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 29420 invoked by uid 500); 29 Dec 2013 23:16:23 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 29412 invoked by uid 99); 29 Dec 2013 23:16:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Dec 2013 23:16:23 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of cryptcom@gmail.com designates 209.85.128.181 as permitted sender) Received: from [209.85.128.181] (HELO mail-ve0-f181.google.com) (209.85.128.181) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Dec 2013 23:16:17 +0000 Received: by mail-ve0-f181.google.com with SMTP id oy12so5622951veb.26 for ; Sun, 29 Dec 2013 15:15:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=RaRU8geU+czYZHSlGvCRFFZinD4VquKkjTpNTHzq1D0=; b=hwCGNRhZZkpYTjgaWQawWYOc5LTWeNi8CCMEqwf2XtmvRagr7E9i9tRp3s0qs8VKA/ o7MUQdb8WzD476wJgSKiRgS8ucyqwTUUotZt3O10Iz4Ij7Y/0g+Q7zjw0zpke/fnCoCA AC9/6wC5yFEzxtwn5fkOsV+Cncz9ouwmHWmq5vt9A4JRyk0CXN0bXhzXBht+YGvsbcJs qb6t2acsaDk+ye94oWWDn8AC5nXh3CNmhMUiuC2zOuUrim1DKyuvi8LUjHHz/Ctq1kTN fdbDmKqxwgNHLkt1QF7thDiAHbKUOH2PTbLyjuh0LB5uCZKWo1H7GrTzJe9mSKplDWZA rsCg== MIME-Version: 1.0 X-Received: by 10.221.33.206 with SMTP id sp14mr179890vcb.29.1388358956043; Sun, 29 Dec 2013 15:15:56 -0800 (PST) Received: by 10.52.105.133 with HTTP; Sun, 29 Dec 2013 15:15:55 -0800 (PST) In-Reply-To: References: <96741685-E5CF-4112-82F7-25A9D1AE158D@gmail.com> Date: Sun, 29 Dec 2013 18:15:55 -0500 Message-ID: Subject: Re: Cassandra unit testing becoming nearly impossible: suggesting alternative. From: Joe Stein To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=001a11365136657a7704eeb482d6 X-Virus-Checked: Checked by ClamAV on apache.org --001a11365136657a7704eeb482d6 Content-Type: text/plain; charset=ISO-8859-1 I updated my repo with Vagrant and bash scripts to install Cassandra 2.0.3 https://github.com/stealthly/scala-cassandra/ 0) git clone https://github.com/stealthly/scala-cassandra 1) cd scala-cassandra 2) vagrant up Cassandra will be running in the virtual machine on 172.16.7.2 and is accessible from your host machine (cqlsh, your app, whatever). To verify step 3 would be ./sbt test just to make sure everything is running right. Everyone time you rebuild the VM (takes a minute or two) it is a whole new instance. If you fork foreground you have to worry about data and that not isolated and other stuff. On Fri, Dec 27, 2013 at 10:48 PM, Edward Capriolo wrote: > I think i will invest the time launching cassandra in a forked forground > process, maybe building the yaml dynamically. > > On Friday, December 27, 2013, Nate McCall wrote: > > I've also moved on to container-based (using Vagrant+docker) setup for > doing automated integration stuff. This is more difficult to configure for > build systems like Jenkins, but it can be done and once completed the > benefits are substantial - as Joe notes, the most immediate is the removal > of variance between different environments. > > However, for in process testing with Maven or similar, the Usergrid > project [0] probably has the most functionally advanced test architecture > [1]. Do understand that it took us a very long time to get there and > involves some fairly tight integration with JUnit and (to a lesser degree) > maven. > > The UG plumbing is purpose built towards a specific data model so it's > not something that can be just dropped in, but it can be pulled apart in a > straight forward way (provided you understand JUnit - which is not really > trivial) and generalized pretty easily. It's all ASF-licensed, so take what > you need if you find it useful. > > [0] https://usergrid.incubator.apache.org/ > > [1] > https://github.com/usergrid/usergrid/blob/master/stack/test-utils/src/main/java/org/usergrid/cassandra/CassandraResource.java > > > > On Wed, Dec 25, 2013 at 2:42 PM, Joe Stein wrote: > > > > I have been using vagrant (e.g. > https://github.com/stealthly/scala-cassandra/ ) which is 100% > reproducible across devs and test systems (prod in some cases). Also have > a Docker setup too https://github.com/pegasussolutions/docker-cassandra . > I have been doing this more and more with clients to better mimic > production before production and smoothing the release process from > development. I also use packer (scripts released soon) to build images too > (http://packer.io) > > Love vagrant, packer and docker!!! Apache Mesos too :) > > > > > > /******************************************* > > Joe Stein > > Founder, Principal Consultant > > Big Data Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop > > ********************************************/ > > > > On Dec 25, 2013, at 3:28 PM, horschi wrote: > > > > Hi Ed, > > > > my opinion on unit testing with C* is: Use the real database, not any > embedded crap :-) > > > > All you need are fast truncates, by which I mean: > > JVM_OPTS="$JVM_OPTS -Dcassandra.unsafesystem=true" > > and > > auto_snapshot: false > > > > This setup works really nice for me (C* 1.1 and 1.2, have not tested 2.0 > yet). > > > > Imho this setup is better for multiple reasons: > > - No extra classpath issues > > - Faster: Running JUnits and C* in one JVM would require a really large > heap (for me at least). > > - Faster: No Cassandra startup everytime I run my tests. > > > > The only downside is that developers must change the properties in their > configs. > > > > cheers, > > Christian > > > > > > > > On Tue, Dec 24, 2013 at 9:31 PM, Edward Capriolo > wrote: > > > > I am not sure there how many people have been around developing > Cassandra for as long as I have, but the state of all the client libraries > and the cassandra server is WORD_I_DONT_WANT_TO_SAY. > > Here is an example of something I am seeing: > > ERROR 14:59:45,845 Exception in thread Thread[Thrift:5,5,main] > > java.lang.AbstractMethodError: > org.apache.thrift.ProcessFunction.isOneway()Z > > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:51) > > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:194) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:722) > > DEBUG 14:59:51,654 retryPolicy for schema_triggers is 0.99 > > In short: If you are new to cassandra and only using the newest client I > am sure everything is peachy for you. > > For people that have been using Cassandra for a while it is harder to > "jump ship" when something better comes along. You need sometimes to > support both hector and astyanax, it happens. > > For a while I have been using hector. Even not to use hector as an API, > but the one nice thing I got from hector was a simple EmbeddedServer that > would clean up after itself. Hector seems badly broken at the moment. I > have no idea how the current versions track with anything out there in the > cassandra world. > > For a while I played with https://github.com/Netflix/astyanax, which > has it's own version and schemes and dependent libraries. (astyanax has > some packaging error that forces me into maven3 > > > > -- > > ----------------- > > Nate McCall > > Austin, TX > > @zznate > > > > Co-Founder & Sr. Technical Consultant > > Apache Cassandra Consulting > > http://www.thelastpickle.com > > -- > Sorry this was sent from mobile. Will do less grammar and spell check than > usual. > --001a11365136657a7704eeb482d6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I updated my repo with Vagrant and bash scripts to in= stall Cassandra 2.0.3 https://github.com/stealthly/scala-cassandra/

0) git clone = https://github.com= /stealthly/scala-cassandra=A0
1) cd scala-cassandra=A0
2) vagrant up=A0

Cassandra will be run= ning in the virtual machine on 172.16.7.2 and is accessible from your host = machine (cqlsh, your app, whatever).

To verify step 3 would be ./sbt= test just to make sure everything is running right.

Everyone time you rebuild the VM (takes a minute or two) it is a = whole new instance.=A0 If you fork foreground you have to worry about data = and that not isolated and other stuff.

On Fri, Dec 27, 2013 at 10:48 PM, Edward Cap= riolo <edlinuxguru@gmail.com> wrote:
I think i will invest the time launching cassandra in a forked forground pr= ocess, maybe building the yaml dynamically.

On Friday, December 27, 2013, Nate McCall <nate@thelastpickle.com> w= rote:
> I've also moved on to container-based (using Vagrant+docker) setup= for doing automated integration stuff. This is more difficult to configure= for build systems like Jenkins, but it can be done and once completed the = benefits are substantial - as Joe notes, the most immediate is the removal = of variance between different environments.=A0
> However, for in process testing with Maven or similar, the Usergrid pr= oject [0] probably has the most functionally advanced test architecture [1]= . Do understand that it took us a very long time to get there and involves = some fairly tight integration with JUnit and (to a lesser degree) maven.=A0=
> The UG plumbing is purpose built towards a specific data model so it&#= 39;s not something that can be just dropped in, but it can be pulled apart = in a straight forward way (provided you understand JUnit - which is not rea= lly trivial) and generalized pretty easily. It's all ASF-licensed, so t= ake what you need if you find it useful.=A0
> [0]=A0https://usergrid.incubator.apache.org/
> [1]=A0https:/= /github.com/usergrid/usergrid/blob/master/stack/test-utils/src/main/java/or= g/usergrid/cassandra/CassandraResource.java
>
> On Wed, Dec 25, 2013 at 2:42 PM, Joe Stein <cryptcom@gmail.com> wrote:<= br>>
> I have been using vagrant (e.g.=A0https://github.com/steal= thly/scala-cassandra/ ) which is 100% reproducible across devs and test= systems (prod in some cases). =A0Also have a Docker setup too=A0ht= tps://github.com/pegasussolutions/docker-cassandra . =A0I have been doi= ng this more and more with clients to better mimic production before produc= tion and smoothing the release process from development. =A0I also use pack= er (scripts released soon) to build images too (http://packer.io)
> Love vagrant, packer and docker!!! =A0Apache Mesos too :)
>
&= gt;
> /*******************************************
> =A0Joe Ste= in
> =A0Founder, Principal Consultant
> =A0Big Data Open Source= Security LLC
> =A0http://www.stea= lth.ly
> =A0Twitter: @allthingshadoop
> *******************= *************************/
>
> On Dec 25, 2013, at 3:28 PM, hor= schi <horschi@gma= il.com> wrote:
>
> Hi Ed,
>
> my opinion on unit testing with C* is: = Use the real database, not any embedded crap :-)
>
> All you ne= ed are fast truncates, by which I mean:
> JVM_OPTS=3D"$JVM_OPTS = -Dcassandra.unsafesystem=3Dtrue"
> and
> auto_snapshot: false
>
> This setup works real= ly nice for me (C* 1.1 and 1.2, have not tested 2.0 yet).
>
> I= mho this setup is better for multiple reasons:
> - No extra classpath= issues
> - Faster: Running JUnits and C* in one JVM would require a really larg= e heap (for me at least).
> - Faster: No Cassandra startup everytime = I run my tests.
>
> The only downside is that developers must c= hange the properties in their configs.
>
> cheers,
> Christian
>
>
>
> On T= ue, Dec 24, 2013 at 9:31 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
&g= t;
> I am not sure there how many people have been around developing Cassan= dra for as long as I have, but the state of all the client libraries and th= e cassandra server is WORD_I_DONT_WANT_TO_SAY.
> Here is an example of something I am seeing:
> ERROR 14:59:45,84= 5 Exception in thread Thread[Thrift:5,5,main]
> java.lang.AbstractMet= hodError: org.apache.thrift.ProcessFunction.isOneway()Z
> at org.apac= he.thrift.ProcessFunction.process(ProcessFunction.java:51)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.= run(CustomTThreadPoolServer.java:194)
> at java.util.concurrent.Threa= dPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:603)
> at java.lang.Thread.run(Thread.java:722)
> DEBUG= 14:59:51,654 retryPolicy for schema_triggers is 0.99
> In short: If = you are new to cassandra and only using the newest client I am sure everyth= ing is peachy for you.
> For people that have been using Cassandra for a while it is harder to = "jump ship" when something better comes along. You need sometimes= to support both hector and astyanax, it happens.=A0
> For a while I = have been using hector. Even not to use hector as an API, but the one nice = thing I got from hector was a simple EmbeddedServer that would clean up aft= er itself. Hector seems badly broken at the moment. I have no idea how the = current versions track with anything out there in the cassandra world.=A0 > For a while I played with https://github.com/Netflix/astyanax, which has it= 's own version and schemes and dependent libraries. (astyanax has some = packaging error that forces me into maven3
>
> --
> -----= ------------
> Nate McCall
> Austin, TX
> @zznate
>=
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra= Consulting
> http://www.= thelastpickle.com

--
Sorry this = was sent from mobile. Will do less grammar and spell check than usual.

--001a11365136657a7704eeb482d6--