cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <>
Subject Re: Discussion: release quality
Date Wed, 30 Nov 2011 03:14:52 GMT
I need at least a week, maybe two to promote anything to staging which is mainly because we
do weekly releases.   I could introduce a 2 day turn around but only with a more fixed type
schedule.  I am running 0.8.6 in production and REALLY want to upgrade for nothing more than
getting compression ( the cost of petabytes of uncompressed data is just stupid ).  So however
I can help in changing my process OR better understanding the PMC here I am game for.  

One thing I use C* for is holding days worth of data and re-running those days for regression
on our software... simulating production... It might not take much to reverse it.

Joe Stein
Twitter: @allthingshadoop

On Nov 29, 2011, at 10:04 PM, Edward Capriolo <> wrote:

> On Tue, Nov 29, 2011 at 6:16 PM, Jeremy Hanna <>wrote:
>> I'd like to start a discussion about ideas to improve release quality for
>> Cassandra.  Specifically I wonder if the community can do more to help the
>> project as a whole become more solid.  Cassandra has an active and vibrant
>> community using Cassandra for a variety of things.  If we all pitch in a
>> little bit, it seems like we can make a difference here.
>> Release quality is difficult, especially for a distributed system like
>> Cassandra.  The core devs have done an amazing job with this considering
>> how complicated it is.  Currently, there are several things in place to
>> make sure that a release is generally usable:
>> - review-then-commit
>> - 72 hour voting period
>> - at least 3 binding +1 votes
>> - unit tests
>> - integration tests
>> Then there is the personal responsibility aspect - testing a release in a
>> staging environment before pushing it to production.
>> I wonder if more could be done here to give more confidence in releases.
>> I wanted to see if there might be ways that the community could help out
>> without being too burdensome on either the core devs or the community.
>> Some ideas:
>> More automation: run YCSB and stress with various setups.  Maybe people
>> can rotate donating cloud instances (or simply money for them) but have a
>> common set of scripts to do this in the source.
>> Dedicated distributed test suite: I know there has been work done on
>> various distributed test suites (which is great!) but none have really
>> caught on so far.
>> I know what the apache guidelines say, but what if the community could
>> help out with the testing effort in a more formal way.  For example, for
>> each release to be finalized, what if there needed to be 3 community
>> members that needed to try it out in their own environment?
>> What if there was a post release +1 vote for the community to sign off on
>> - sort of a "works for me" kind of thing to reassure others that it's safe
>> to try.  So when the release email gets posted to the user list, start a
>> tradition of people saying +1 in reply if they've tested it out and it
>> works for them.  That's happening informally now when there are problems,
>> but it might be nice to see a vote of confidence.  Just another idea.
>> Any other ideas or variations?
> I am no software engineering guru, but whenever I +1 a hive release I
> actually do checkout the code and run a couple queries. Mostly I find that
> because there is just so many things not unit testable like those gosh darn
> bash scripts that launch Java applications. There have been times when even
> after multiple patch revisions and passing unit tests something just does
> not work in the real world. So I never +1 a binary release I don't spend an
> hour with and if possible I try twisting the knobs on any new feature or at
> least just trying the basics.Hive is aiming for something like quarterly
> releases.
> So possibly better to have Cassandra do time based releases. It does not
> have to be quarterly but if people want bleeding edge features (something
> committed 2 days ago) really they should go out and build something from
> trunk.
> It seems like Cassandra devs have the voting and releasing down to a
> science but from my world the types of bugs I worry about are data file
> corruption, and any weird bug that would result in data faults like
> read_repair not working or writes not going to the write nodes, or bloom
> filters giving a faulty result. New features are great and I love seeing
> them but I can wait for those.
> Updates now even trivial ones get political, you just never want to be the
> guy that champions a update and then not have it go well :)
> Most users of Cassandra are going to have large clusters and really the
> project should not outstrip the common users ability to stay up to date.
> You have to figure that a large cluster like 20 nodes with maybe 200Gb
> data/node, doing a rolling restart without degrading performance is going
> to take some time. This is more then 'yum update cassandra'
> /etc/init.d/cassandra restart' and with risk of something going wrong
> people need time to QA and time for ops. This type of person does not like
> to fall many releases behind and likewise can not be updating too often
> either.
> I have never had to roll back a release but I do wait usually for a month
> before running one to make sure there is not following soon.

View raw message