incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jahangir Mohammed <md.jahangi...@gmail.com>
Subject Re: Pending ReadStage is exploding on only one node
Date Mon, 21 Nov 2011 12:26:50 GMT
I am not so sure from version to version.

1. Which client are you using? Any custom load balancer?
2. Is the hardware on this node any different from other nodes?

Thanks,
Jahangir.

On Mon, Nov 21, 2011 at 5:55 AM, Johann Höchtl <h.hoechtl@ic-drei.de> wrote:

> Hi all,
>
> I'm experiencing strange behaviour of my 6-node cassandra cluster and I
> hope some one can explain, what I'm doing wrong.
>
> The setting:
> 6-Cassandra Nodes 1.0.3
> Random Partitioning
> The ColumnFamily in question has a replication factor of 2 and stores
> products of different shops with a secondary index on shop_id.
>
> Twice a day, I do an update of the data with the following mechanism:
> Get all keys of a shop.
> Read the new CSV.
> Insert the rows from the csv, which keys are not present and delete the
> rows which are not longer present.
> Update all prices of the products from the csv and set an update_date.
>
> I'm measuring a high load value on a few nodes during the update process
> (which is normal), but one node keeps the high load after the process for a
> long time.
> I checked the tpstats and found out, that on this node there are over 50k
> pending ReadStage tasks.
> All the other nodes don't have that behaviour.
>
> I already had this problem on cassandra 0.7, but after upgrading to 0.8 it
> disappeared. Now it is back.
>
> Any suggestions?
>
> Thanks,
> Hans
>

Mime
View raw message