Yes, we saw this same behavior.

A couple of months ago, we moved a large portion of our data out of Postgres and into Cassandra.  The initial migration was done in a "distributed" manner:  we had 600 (or 800, can't remember) processes reading from Postgres and writing to Cassandra in tight loops.  This caused the exact behavior you described.  We also did a read before a write.

After we got through the initial data migration, our normal workload is *much* less writes (and reads for that matter) such that our cluster can easily handle it, so we didn't investigate further.

-- C


On Sat, Nov 23, 2013 at 10:55 PM, srmore <comomore@gmail.com> wrote:
Hello,
We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap bloom filters and other improvements.

We see a lot of messages dropped under high load conditions. We noticed that when we do heavy read AND write simultaneously (we read first and check whether the key exists if not we write it) Cassandra heap increases dramatically and then gossip marks the node down (as a result of high load on the node).


Under heavy 'reads only' we don't see this behavior.  Has anyone seen this behavior ? any suggestions.

Thanks !