incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bialecki <andrew.biale...@gmail.com>
Subject High number of ReplicateOnWriteStage All timed blocked, counter CF
Date Tue, 22 Oct 2013 08:23:59 GMT
Hey everyone,

We're stress testing writes for a few counter CFs and noticed one one node
we got to the point where the ReplicateOnWriteStage thread pool was backed
up and it started blocking those tasks. This cluster is six nodes, RF=3,
running 1.2.9. All CFs have LCS with 160 MB sstables. All writes were
CL.ONE.

Few questions:

   1. What causes a RoW (replicate of write) task to be blocked? The queue
   maxes out at 4128, which seems to be 32 * (128 + 1). 32 is the number of
   concurrent_writers we have.

   2. Given this is a counter CF, can those dropped RoWs be repaired with a
   "nodetool repair?" From my understanding of how counter writes work, until
   we run that repair, if we're not using CL.ALL / read_repair_chance = 1, we
   will get some incorrect reads, but a repair will fix things. Is that right?

   3. The CPU on the node where we started seeing the number of blocked
   tasks increase was pegged, but I/O was not saturated. There were
   compactions running on those column families as well. Is there a setting we
   could consider altering that might prevent that back up or is the answer
   likely, "increase the number of nodes to get more throughput."


Thanks in advance for any insights!

Andrew

Mime
View raw message