cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Bijnens <nat...@nathan.gs>
Subject ReadStage high number of pending tasks
Date Thu, 09 Jul 2015 11:09:53 GMT
We are having issues where one node out of 6 shows increasing ReadStage
pending tasks, for all others this is close to 0.

[image: graph.png]

When connecting VisualVM to Cassandra we see that the following method
takes all cpu time, this method is called from all the ReadStage threads:
org.apache.cassandra.db.RangeTombstoneList.addInternal(RangeTombstoneList.java:545)

[image: Capture.PNG]

Because the ReadStage pending is so high the node almost stops responding.

The cpu usage on this node is much higher than on all the other ones:
[image: stacked.png]
There is also almost no disk IO going on.

When restarting cassandra the load drops for a second and rises again. The
ReadStage pending tasks reset to 0, but rice fast again.

The following settings are configured:
read_request_timeout_in_ms: 60000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 10000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000

Anyone seen similar behaviour?

We are using Cassandra 2.0.14.

Thanks,
  Nathan

Mime
View raw message