Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of andrew.bialecki@gmail.com
 designates 209.85.212.182 as permitted sender)
MIME-Version: 1.0
Date: Tue, 22 Oct 2013 04:23:59 -0400
Message-ID: 
 <CAFDWQMQo3z_wUT2RgmR-y59_ADvQZLsZ8mu-vX=xN3F3AxT5NA@mail.gmail.com>
Subject: High number of ReplicateOnWriteStage All timed blocked, counter CF
From: Andrew Bialecki <andrew.bialecki@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7b5d57ce626ddf04e9501f70

--047d7b5d57ce626ddf04e9501f70
Content-Type: text/plain; charset=ISO-8859-1

Hey everyone,

We're stress testing writes for a few counter CFs and noticed one one node
we got to the point where the ReplicateOnWriteStage thread pool was backed
up and it started blocking those tasks. This cluster is six nodes, RF=3,
running 1.2.9. All CFs have LCS with 160 MB sstables. All writes were
CL.ONE.

Few questions:

   1. What causes a RoW (replicate of write) task to be blocked? The queue
   maxes out at 4128, which seems to be 32 * (128 + 1). 32 is the number of
   concurrent_writers we have.

   2. Given this is a counter CF, can those dropped RoWs be repaired with a
   "nodetool repair?" From my understanding of how counter writes work, until
   we run that repair, if we're not using CL.ALL / read_repair_chance = 1, we
   will get some incorrect reads, but a repair will fix things. Is that right?

   3. The CPU on the node where we started seeing the number of blocked
   tasks increase was pegged, but I/O was not saturated. There were
   compactions running on those column families as well. Is there a setting we
   could consider altering that might prevent that back up or is the answer
   likely, "increase the number of nodes to get more throughput."


Thanks in advance for any insights!

Andrew

--047d7b5d57ce626ddf04e9501f70
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hey everyone,<div><br></div><div>We&#39;re stress testing =
writes for a few counter CFs and noticed one one node we got to the point w=
here the ReplicateOnWriteStage thread pool was backed up and it started blo=
cking those tasks. This cluster is six nodes, RF=3D3, running 1.2.9. All CF=
s have LCS with 160 MB sstables. All writes were CL.ONE.</div>
<div><br></div><div>Few questions:</div><div><ol><li>What causes a RoW (rep=
licate of write) task to be blocked? The queue maxes out at 4128, which see=
ms to be 32 * (128 + 1). 32 is the number of concurrent_writers we have.<br=
>
<br></li><li>Given this is a counter CF, can those dropped RoWs be repaired=
 with a &quot;nodetool repair?&quot; From my understanding of how counter w=
rites work, until we run that repair, if we&#39;re not using CL.ALL / read_=
repair_chance =3D 1, we will get some incorrect reads, but a repair will fi=
x things. Is that right?<br>
<br></li><li>The CPU on the node where we started seeing the number of bloc=
ked tasks increase was pegged, but I/O was not saturated. There were compac=
tions running on those column families as well. Is there a setting we could=
 consider altering that might prevent that back up or is the answer likely,=
 &quot;increase the number of nodes to get more throughput.&quot;<br>
</li></ol><div><br></div><div>Thanks in advance for any insights!</div></di=
v><div><br></div><div>Andrew</div></div>

--047d7b5d57ce626ddf04e9501f70--