cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Hobbs <ty...@datastax.com>
Subject Re: Consistency model
Date Sat, 16 Apr 2011 21:20:25 GMT
James,

Would you mind sharing your reader process code as well?

On Fri, Apr 15, 2011 at 1:14 PM, James Cipar <jcipar@cmu.edu> wrote:

> I've been experimenting with the consistency model of Cassandra, and I
> found something that seems a bit unexpected.  In my experiment, I have 2
> processes, a reader and a writer, each accessing a Cassandra cluster with a
> replication factor greater than 1.  In addition, sometimes I generate
> background traffic to simulate a busy cluster by uploading a large data file
> to another table.
>
> The writer executes a loop where it writes a single row that contains just
> an sequentially increasing sequence number and a timestamp.  In python this
> looks something like:
>
>    while time.time() < start_time + duration:
>        target_server = random.sample(servers, 1)[0]
>        target_server = '%s:9160'%target_server
>
>        row = {'seqnum':str(seqnum), 'timestamp':str(time.time())}
>        seqnum += 1
>        # print 'uploading to server %s, %s'%(target_server, row)
>
>        pool = pycassa.connect('Keyspace1', [target_server])
>        cf = pycassa.ColumnFamily(pool, 'Standard1')
>        cf.insert('foo', row, write_consistency_level=consistency_level)
>        pool.dispose()
>
>        if sleeptime > 0.0:
>            time.sleep(sleeptime)
>
>
> The reader simply executes a loop reading this row and reporting whenever a
> sequence number is *less* than the previous sequence number.  As expected,
> with consistency_level=ConsistencyLevel.ONE there are many inconsistencies,
> especially with a high replication factor.
>
> What is unexpected is that I still detect inconsistencies when it is set at
> ConsistencyLevel.QUORUM.  This is unexpected because the documentation seems
> to imply that QUORUM will give consistent results.  With background traffic
> the average difference in timestamps was 0.6s, and the maximum was >3.5s.
>  This means that a client sees a version of the row, and can subsequently
> see another version of the row that is 3.5s older than the previous.
>
> What I imagine is happening is this, but I'd like someone who knows that
> they're talking about to tell me if it's actually the case:
>
> I think Cassandra is not using an atomic commit protocol to commit to the
> quorum of servers chosen when the write is made.  This means that at some
> point in the middle of the write, some subset of the quorum have seen the
> write, while others have not.  At this time, there is a quorum of servers
> that have not seen the update, so depending on which quorum the client reads
> from, it may or may not see the update.
>
> Of course, I understand that the client is not *choosing* a bad quorum to
> read from, it is just the first `q` servers to respond, but in this case it
> is effectively random and sometimes an bad quorum is "chosen".
>
> Does anyone have any other insight into what is going on here?




-- 
Tyler Hobbs
Software Engineer, DataStax <http://datastax.com/>
Maintainer of the pycassa <http://github.com/pycassa/pycassa> Cassandra
Python client library

Mime
View raw message