Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of hosschao@gmail.com designates
 209.85.210.172 as permitted sender)
MIME-Version: 1.0
Date: Tue, 24 Apr 2012 01:48:58 +0800
Message-ID: 
 <CAGg-zQV1qK=VqKQdiYfz_uRM=yw4tYvoEdpCaQAQVhKRMkG3zw@mail.gmail.com>
Subject: High latency of cassandra
From: =?GB2312?B?wu2zrA==?= <hosschao@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=e89a8f2352f5ae5a8604be5c4019

--e89a8f2352f5ae5a8604be5c4019
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I have some troubles of cassandra in my production:

I build up a RPC server which using hector client to manipulate the
cassandra. Wired things happen nowadays: the latency of RPC sometimes
becames very high (10seconds~70seconds) in several minutes and reduce
to normal level (30ms in average) after that time. I investigate the
debug log of cassandra. During high latency time, the cassandra output
lots of message like:
"IncomingTcpConnection.java(116) Version is now 3. "
Seems everything be blocked during that time.

Our settings as following:
The version of cassandra is 1.0.1 and hector version is 0.7.0 for the
compatible of thrift version which we use (0.5.0)
The cluster contains 4 nodes and all of them are seeds. The
gcgraceseconds is 0 since we needn't delete the data

p.s. It works well for a long time (3 months) but becames crazy these
days after we push the new RPC server which supports bigger data
saving (2mb in average). I'm not sure if these is the reason.

Hope getting your replay~~

Thanks,

Chao.

--e89a8f2352f5ae5a8604be5c4019
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<span style>Hi all,</span><br style><br style><span style>I have some troub=
les of cassandra in my production:</span><br style><br style><span style>I =
build up a RPC server which using hector client to manipulate the</span><br=
 style>
<span style>cassandra. Wired things happen nowadays: the latency of RPC som=
etimes</span><br style><span style>becames very high (10seconds~70seconds) =
in several minutes and reduce</span><br style><span style>to normal level (=
30ms in average) after that time. I investigate the</span><br style>
<span style>debug log of cassandra. During high latency time, the cassandra=
 output</span><br style><span style>lots of message like:</span><br style><=
span style>&quot;IncomingTcpConnection.java(</span><span style>116) Version=
 is now 3. &quot;</span><br style>
<span style>Seems everything be blocked during that time.</span><br style><=
br style><span style>Our settings as following:</span><br style><span style=
>The version of cassandra is 1.0.1 and hector version is 0.7.0 for the</spa=
n><br style>
<span style>compatible of thrift version which we use (0.5.0)</span><br sty=
le><span style>The cluster contains 4 nodes and all of them are seeds. The<=
/span><br style><span style>gcgraceseconds is 0 since we needn&#39;t delete=
 the data</span><br style>
<br style><span style>p.s. It works well for a long time (3 months) but bec=
ames crazy these</span><br style><span style>days after we push the new RPC=
 server which supports bigger data</span><br style><span style>saving (2mb =
in average). I&#39;m not sure if these is the reason.</span><br style>
<br style><span style>Hope getting your replay~~</span><br style><br style>=
<span style>Thanks,</span><br style><br style><span style>Chao.</span><br s=
tyle>

--e89a8f2352f5ae5a8604be5c4019--