cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Gallamore <mike.e.gallam...@googlemail.com>
Subject writes to Cassandra failing occasionally
Date Wed, 07 Apr 2010 20:19:26 GMT
I have writes to cassandra that are failing, or at least a read shortly 
after a write is still getting an old value. I realize Cassandra is 
"eventually consistent" but this system is a single CPU single node with 
consistency level set to 1, so this seems odd to me.

My setup:

Cassandra 0.6rc1

client: writing using Net::Cassandra::Easy 0.09 (perl thrift interface)

Cluster: single node. Default out of the box setup. Single CPU system 
(its a P4 dinosaur). The client library defaults to read and write 
consistency of 1 which is the number of nodes so is like "all" in my case.

I have a test that modifies data and then tests to see if they have the 
expected value. These tests pass sometimes and sometimes fail part way 
through the test suite. This doesn't seem to be predictable, stopping 
cassandra, nuking the data directory and restarting doesn't make the 
system any more likely to pass. Passing or failing the test one time 
doesn't seem to correlate with the pass or failure of the next attempt. 
What I'm seeing is old data still being returned to get requests after 
the mutate has run.

For example: I modified the "set" method of my program to output the 
value being saved, the old value and the new value and I see things like 
this when the tests fail:

Expected value was
55, but got
72 instead, previous value was
72 at lib/ET/Storage/Cassandra.pm line 96.

This means the previous value was 72, the modifications that where done 
should have resulted in the score being 55 but I got a 72 instead. Any 
ideas what could cause this?

As an aside I motified some other code to use Net::Cassandra instead of 
Net::Cassandra::Easy and noticed that it seems to run 3-4X slower. Both 
aren't stunningly fast. The test clients are running on the same machine 
as Cassandra, and I'm only getting somewhere between 100-400 (huge 
variance) with N::C::Easy and 30-90 with N::C. This test is writing key 
value pairs, with the keys being an incrementing numbber, and the values 
being a log line from one of our systems (~200 character string). I'm 
surprised there is such a huge difference in speed between the two 
modules and that the transactions per second are so low even on my 
3.2Ghz P4 2GB RAM box. I tried dropping the consistency level down to 
zero but it had a negligible affect.






Mime
View raw message