incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey <alexe...@gmail.com>
Subject Re: Increased Cassandra connection latency
Date Mon, 26 May 2014 07:25:13 GMT
It looks like a memory leak to me. All Cassandra processes are with the
following JVM settings configured: -Xms10240M -Xmx10240M -Xmn2400M. The
OldGen reaches nearly the maximum and it looks like the oldgen objects not
being garbage collected.

Alexey


On Fri, May 23, 2014 at 3:56 PM, Nate McCall <nate@thelastpickle.com> wrote:

> Is compaction keeping up? Those replication settings are high - for every
> write, 10 nodes are doing something).
>
> What other monitoring stats do you have - what is IO, CPU and network
> traffic like? Is the JVM GC activity growing?
>
> Anything else stick out like growing number of network connections to 9160
> or 9042 on the cluster?
>
> The only leak-y things I've seen lately could either be:
>
>  from a JVM bug leaking permgen from JMX invocation:
> https://issues.apache.org/jira/browse/CASSANDRA-6541
>
> or if you have anything using thrift (this includes opscenter) and you are
> using HSHA as the rpc_server_type:
> https://issues.apache.org/jira/browse/CASSANDRA-4265
>
> Otherwise, I think this is config/setup tuning.
>
>
>
>
> On Fri, May 23, 2014 at 7:00 AM, Alexey <alexejka@gmail.com> wrote:
>
>> Hi all,
>>
>> I've noticed increased latency on our tomcat REST-service (average 30ms,
>> max > 2sec). We are using Cassandra 1.2.16 with official DataStax Java
>> driver v1.0.3.
>>
>> Our setup:
>>
>> * 2 DCs
>> * each DC: 7 nodes
>> * RF=5
>> * Leveled compaction
>>
>> After cassandra restart on all nodes, the latencies are alright again
>> (average < 5ms, max 50ms).
>>
>> Any thoughts are greatly appreciated.
>>
>> Thanks,
>> Alexey
>>
>
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Mime
View raw message