Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of
 alexey.sverdelov@googlemail.com designates 209.85.223.179 as permitted
 sender)
MIME-Version: 1.0
Sender: alexey.sverdelov@googlemail.com
In-Reply-To: 
 <CAKmMYa_vhCLZwzC4_ZnuDzVkP0HvcY7L41MkTDBTeGvk86+PuA@mail.gmail.com>
References: 
 <CAJAh7M=Esv3ZuaAQLNqZSWg2A=dTK82EiTmh4gYb70AiCkEABw@mail.gmail.com>
 <CAKmMYa_vhCLZwzC4_ZnuDzVkP0HvcY7L41MkTDBTeGvk86+PuA@mail.gmail.com>
From: Alexey <alexejka@gmail.com>
Date: Mon, 26 May 2014 09:25:13 +0200
Message-ID: 
 <CAJAh7MkrFwskQjnX-eYAgeX4Nj7tCzqnzv0GUCcY7maNmQud9w@mail.gmail.com>
Subject: Re: Increased Cassandra connection latency
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7bd752661f3be004fa487c1c

--047d7bd752661f3be004fa487c1c
Content-Type: text/plain; charset=UTF-8

It looks like a memory leak to me. All Cassandra processes are with the
following JVM settings configured: -Xms10240M -Xmx10240M -Xmn2400M. The
OldGen reaches nearly the maximum and it looks like the oldgen objects not
being garbage collected.

Alexey


On Fri, May 23, 2014 at 3:56 PM, Nate McCall <nate@thelastpickle.com> wrote:

> Is compaction keeping up? Those replication settings are high - for every
> write, 10 nodes are doing something).
>
> What other monitoring stats do you have - what is IO, CPU and network
> traffic like? Is the JVM GC activity growing?
>
> Anything else stick out like growing number of network connections to 9160
> or 9042 on the cluster?
>
> The only leak-y things I've seen lately could either be:
>
>  from a JVM bug leaking permgen from JMX invocation:
> https://issues.apache.org/jira/browse/CASSANDRA-6541
>
> or if you have anything using thrift (this includes opscenter) and you are
> using HSHA as the rpc_server_type:
> https://issues.apache.org/jira/browse/CASSANDRA-4265
>
> Otherwise, I think this is config/setup tuning.
>
>
>
>
> On Fri, May 23, 2014 at 7:00 AM, Alexey <alexejka@gmail.com> wrote:
>
>> Hi all,
>>
>> I've noticed increased latency on our tomcat REST-service (average 30ms,
>> max > 2sec). We are using Cassandra 1.2.16 with official DataStax Java
>> driver v1.0.3.
>>
>> Our setup:
>>
>> * 2 DCs
>> * each DC: 7 nodes
>> * RF=5
>> * Leveled compaction
>>
>> After cassandra restart on all nodes, the latencies are alright again
>> (average < 5ms, max 50ms).
>>
>> Any thoughts are greatly appreciated.
>>
>> Thanks,
>> Alexey
>>
>
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

--047d7bd752661f3be004fa487c1c
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>It looks like a memory leak to me. All Cassandra proc=
esses are with the following JVM settings configured: -Xms10240M -Xmx10240M=
 -Xmn2400M. The OldGen reaches nearly the maximum and it looks like the old=
gen objects not being garbage collected.<br>

<br></div><div>Alexey<br></div></div><div class=3D"gmail_extra"><br><br><di=
v class=3D"gmail_quote">On Fri, May 23, 2014 at 3:56 PM, Nate McCall <span =
dir=3D"ltr">&lt;<a href=3D"mailto:nate@thelastpickle.com" target=3D"_blank"=
>nate@thelastpickle.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>Is compaction keeping =
up? Those replication settings are high - for every write, 10 nodes are doi=
ng something).=C2=A0</div>

<div><br></div><div>What other monitoring stats do you have - what is IO, C=
PU and network traffic like? Is the JVM GC activity growing?</div>
<div><br></div><div>Anything else stick out like growing number of network =
connections to 9160 or 9042 on the cluster?=C2=A0<br></div><div><br></div>T=
he only leak-y things I&#39;ve seen lately could either be:=C2=A0<div><br><=
/div>

<div>
from a JVM bug leaking permgen from JMX invocation:<div><a href=3D"https://=
issues.apache.org/jira/browse/CASSANDRA-6541" target=3D"_blank">https://iss=
ues.apache.org/jira/browse/CASSANDRA-6541</a><br></div><div><br></div><div>

or if you have anything using thrift (this includes opscenter) and you are =
using HSHA as the rpc_server_type:</div>
<div><a href=3D"https://issues.apache.org/jira/browse/CASSANDRA-4265" targe=
t=3D"_blank">https://issues.apache.org/jira/browse/CASSANDRA-4265</a><br></=
div><div><br></div><div>Otherwise, I think this is config/setup tuning.=C2=
=A0</div>

<div><br></div>
<div><br></div></div></div><div class=3D"gmail_extra"><div><div class=3D"h5=
"><br><br><div class=3D"gmail_quote">On Fri, May 23, 2014 at 7:00 AM, Alexe=
y <span dir=3D"ltr">&lt;<a href=3D"mailto:alexejka@gmail.com" target=3D"_bl=
ank">alexejka@gmail.com</a>&gt;</span> wrote:<br>


<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div>Hi all,=
<br><br>I&#39;ve noticed increased latency on our
 tomcat REST-service (average 30ms, max &gt; 2sec). We are using=20
Cassandra 1.2.16 with official DataStax Java driver v1.0.3. <br><br>
</div>Our setup:<br><br></div>* 2 DCs<br></div><div>* each DC: 7 nodes<br><=
/div>* RF=3D5<br></div>* Leveled compaction<br><div><br><span>After cassand=
ra restart on all nodes, the latencies are alright again (average &lt; 5ms,=
 max 50ms).</span><br>


<br>
Any thoughts are greatly appreciated.<br><br></div><div>Thanks,<br></div>Al=
exey</div>
</blockquote></div><br><br clear=3D"all"><div><br></div></div></div><span c=
lass=3D"HOEnZb"><font color=3D"#888888">-- <br><div dir=3D"ltr">-----------=
------<br>Nate McCall<br>Austin, TX<br>@zznate<br><br>Co-Founder &amp; Sr. =
Technical Consultant<br>

Apache Cassandra Consulting<br><a href=3D"http://www.thelastpickle.com" tar=
get=3D"_blank">http://www.thelastpickle.com</a></div>

</font></span></div>
</blockquote></div><br></div>

--047d7bd752661f3be004fa487c1c--