Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of md.jahangir27@gmail.com
 designates 209.85.161.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4ECA2E22.9020801@ic-drei.de>
References: <4ECA2E22.9020801@ic-drei.de>
Date: Mon, 21 Nov 2011 07:26:50 -0500
Message-ID: 
 <CANNh1xq1Vbn=Fs1QC6F9MkC8zBNxObxtE07o2kZshfY_S8Z0kQ@mail.gmail.com>
Subject: Re: Pending ReadStage is exploding on only one node
From: Jahangir Mohammed <md.jahangir27@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=f46d04083d9f1c830004b23dcd04

--f46d04083d9f1c830004b23dcd04
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I am not so sure from version to version.

1. Which client are you using? Any custom load balancer?
2. Is the hardware on this node any different from other nodes?

Thanks,
Jahangir.

On Mon, Nov 21, 2011 at 5:55 AM, Johann H=F6chtl <h.hoechtl@ic-drei.de> wro=
te:

> Hi all,
>
> I'm experiencing strange behaviour of my 6-node cassandra cluster and I
> hope some one can explain, what I'm doing wrong.
>
> The setting:
> 6-Cassandra Nodes 1.0.3
> Random Partitioning
> The ColumnFamily in question has a replication factor of 2 and stores
> products of different shops with a secondary index on shop_id.
>
> Twice a day, I do an update of the data with the following mechanism:
> Get all keys of a shop.
> Read the new CSV.
> Insert the rows from the csv, which keys are not present and delete the
> rows which are not longer present.
> Update all prices of the products from the csv and set an update_date.
>
> I'm measuring a high load value on a few nodes during the update process
> (which is normal), but one node keeps the high load after the process for=
 a
> long time.
> I checked the tpstats and found out, that on this node there are over 50k
> pending ReadStage tasks.
> All the other nodes don't have that behaviour.
>
> I already had this problem on cassandra 0.7, but after upgrading to 0.8 i=
t
> disappeared. Now it is back.
>
> Any suggestions?
>
> Thanks,
> Hans
>

--f46d04083d9f1c830004b23dcd04
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I am not so sure from version to version.<br><br>1. Which client are you us=
ing? Any custom load balancer?<br>2. Is the hardware on this node any diffe=
rent from other nodes?<br><br>Thanks,<br>Jahangir.<br><br><div class=3D"gma=
il_quote">
On Mon, Nov 21, 2011 at 5:55 AM, Johann H=F6chtl <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:h.hoechtl@ic-drei.de">h.hoechtl@ic-drei.de</a>&gt;</span> wro=
te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex;">
Hi all,<br>
<br>
I&#39;m experiencing strange behaviour of my 6-node cassandra cluster and I=
 hope some one can explain, what I&#39;m doing wrong.<br>
<br>
The setting:<br>
6-Cassandra Nodes 1.0.3<br>
Random Partitioning<br>
The ColumnFamily in question has a replication factor of 2 and stores produ=
cts of different shops with a secondary index on shop_id.<br>
<br>
Twice a day, I do an update of the data with the following mechanism:<br>
Get all keys of a shop.<br>
Read the new CSV.<br>
Insert the rows from the csv, which keys are not present and delete the row=
s which are not longer present.<br>
Update all prices of the products from the csv and set an update_date.<br>
<br>
I&#39;m measuring a high load value on a few nodes during the update proces=
s (which is normal), but one node keeps the high load after the process for=
 a long time.<br>
I checked the tpstats and found out, that on this node there are over 50k p=
ending ReadStage tasks.<br>
All the other nodes don&#39;t have that behaviour.<br>
<br>
I already had this problem on cassandra 0.7, but after upgrading to 0.8 it =
disappeared. Now it is back.<br>
<br>
Any suggestions?<br>
<br>
Thanks,<br>
Hans<br>
</blockquote></div><br>

--f46d04083d9f1c830004b23dcd04--