incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johann Höchtl <h.hoec...@ic-drei.de>
Subject Pending ReadStage is exploding on only one node
Date Mon, 21 Nov 2011 10:55:30 GMT
Hi all,

I'm experiencing strange behaviour of my 6-node cassandra cluster and I 
hope some one can explain, what I'm doing wrong.

The setting:
6-Cassandra Nodes 1.0.3
Random Partitioning
The ColumnFamily in question has a replication factor of 2 and stores 
products of different shops with a secondary index on shop_id.

Twice a day, I do an update of the data with the following mechanism:
Get all keys of a shop.
Read the new CSV.
Insert the rows from the csv, which keys are not present and delete the 
rows which are not longer present.
Update all prices of the products from the csv and set an update_date.

I'm measuring a high load value on a few nodes during the update process 
(which is normal), but one node keeps the high load after the process 
for a long time.
I checked the tpstats and found out, that on this node there are over 
50k pending ReadStage tasks.
All the other nodes don't have that behaviour.

I already had this problem on cassandra 0.7, but after upgrading to 0.8 
it disappeared. Now it is back.

Any suggestions?

Thanks,
Hans

Mime
View raw message