Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of h.hoechtl@ic-drei.de designates
 188.40.90.211 as permitted sender)
Message-ID: <4ECA2E22.9020801@ic-drei.de>
Date: Mon, 21 Nov 2011 11:55:30 +0100
From: =?ISO-8859-15?Q?Johann_H=F6chtl?= <h.hoechtl@ic-drei.de>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.23) Gecko/20110921 Thunderbird/3.1.15
MIME-Version: 1.0
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Pending ReadStage is exploding on only one node
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit

Hi all,

I'm experiencing strange behaviour of my 6-node cassandra cluster and I 
hope some one can explain, what I'm doing wrong.

The setting:
6-Cassandra Nodes 1.0.3
Random Partitioning
The ColumnFamily in question has a replication factor of 2 and stores 
products of different shops with a secondary index on shop_id.

Twice a day, I do an update of the data with the following mechanism:
Get all keys of a shop.
Read the new CSV.
Insert the rows from the csv, which keys are not present and delete the 
rows which are not longer present.
Update all prices of the products from the csv and set an update_date.

I'm measuring a high load value on a few nodes during the update process 
(which is normal), but one node keeps the high load after the process 
for a long time.
I checked the tpstats and found out, that on this node there are over 
50k pending ReadStage tasks.
All the other nodes don't have that behaviour.

I already had this problem on cassandra 0.7, but after upgrading to 0.8 
it disappeared. Now it is back.

Any suggestions?

Thanks,
Hans