cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bialecki <>
Subject Lots of replicate on write tasks pending, want to investigate
Date Wed, 03 Jul 2013 16:59:31 GMT
In one of our load tests, we're incrementing a single counter column as
well as appending columns to a single row (essentially a timeline). You can
think of it as counting the instances of an event and then keeping a
timeline of those events. The ratio is of increments to "appends" is 1:1.

When we run this on a test cluster with RF = 3, one node gets backed up
with a lot of replicate on write tasks pending, eventually maxing out at
4128. We think it's a disk I/O issue that's causing the slowdown (lot of
reads), but we're still investigating. A few questions that might speed up
understanding the issue:

1. Is there any way to see metadata about the replicate on write tasks
pending? We're splitting apart the load test to pinpoint which of those
operations is causing an issue, but if there's a way to see that queue,
that might save us some work.

2. I'm assuming in our case the cause is incrementing counters because disk
reads are part of the write path for counters and are not for appending
columns to a row. Does that logic make sense?

Thanks in advance,

View raw message