Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of ikatkov@gmail.com designates
 209.85.218.226 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=ZKGAgn2V0HWROjo9L0DPeI13s74bEGYrVBwgKez3E8GEdgcBLQ3MLr7v0e0Qqgpg0M
         FT+eyvNOstPA3DaANw7Br9rCgosCVtYoTb4hYTHzcHJdsbMvIMXFSLVPxDqb9UMz3GwL
         lL7/Jry+nBphxJq9EbxuNjs2Yw6O7a9Ls458E=
MIME-Version: 1.0
From: Igor Katkov <ikatkov@gmail.com>
Date: Thu, 1 Oct 2009 18:51:08 -0400
Message-ID: <23b1e84e0910011551h49a6e8a0xaba1d2d6290c42e6@mail.gmail.com>
Subject: Storage proxy write latency is too high
To: cassandra-user@incubator.apache.org
Content-Type: multipart/alternative; boundary=0015173fe6c6e161870474e77ccb

--0015173fe6c6e161870474e77ccb
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I have the following puzzle:
Storage proxy write latency ~235ms
CF write latency <1 ms

I have 3 nodes in the cluster, Cassandra v.0.4. Tokens evenly distributed.
The client connects to a node and inserts a key with ConsistencyLevel.ONE
If it happen to be a local write operation is fast, same speed as in one
node setup. JMX shows write latency <1 ms
If it happens to be a remote insert StorageProxy sends it to a proper node.
This operation is slow. JMX shows write latency ~ 235ms.
In the same time, on remote node JMX shows same <1ms write latency. So it's
not remote node being sluggish, it's something else.
There are no pending tasks on remote node - JMX counters are always zero,
network is 1Gb, idle. So I can't blame it.


I profiled Cassandra server in JProfiler, could not find a thing. All this
extra time is spent inside QuorumResponseHandler waiting for the condition
to signal. Which should happen as soon as response is received.

There is one pooled TCP connection open to remote host. Hardly a bottleneck,
ThreadPoolExecutors looks OK.

Any ideas why write latency it is so high?

--0015173fe6c6e161870474e77ccb
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,<br><br>I have the following puzzle: <br>Storage proxy write latency ~23=
5ms<br>CF write latency &lt;1 ms<br><br>I have 3 nodes in the cluster, Cass=
andra v.0.4. Tokens evenly distributed. <br>The client connects to a node a=
nd inserts a key with ConsistencyLevel.ONE<br>

If it happen to be a local write operation is fast, same speed as in one no=
de setup. JMX shows write latency &lt;1 ms<br>If it happens to be a remote =
insert StorageProxy sends it to a proper node. This operation is slow. JMX =
shows write latency ~ 235ms. <br>

In the same time, on remote node JMX shows same &lt;1ms write latency. So i=
t&#39;s not remote node being sluggish, it&#39;s something else.<br>There a=
re no pending tasks on remote node - JMX counters are always zero, network =
is 1Gb, idle. So I can&#39;t blame it.<br>

<br><br>I profiled Cassandra server in JProfiler, could not find a thing. A=
ll this extra time is spent inside QuorumResponseHandler waiting for the co=
ndition to signal. Which should happen as soon as response is received.<br>

<br>There is one pooled TCP connection open to remote host. Hardly a bottle=
neck, ThreadPoolExecutors looks OK.<br><br>Any ideas why write latency it i=
s so high?<br>

--0015173fe6c6e161870474e77ccb--