cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Samuelsson <samuelsson.j...@gmail.com>
Subject Extremely long GC
Date Wed, 22 Jan 2014 14:50:58 GMT
Hello,

We've been having problems with long GC pauses and can't seem to get rid of
them.

Our latest test is on a clean machine with Ubuntu 12.04 LTS, Java 1.7.0_45
and JNA installed.
It is a single node cluster with most settings being default, the only
things changed are ip-addresses, cluster name and partitioner (to
RandomPartitioner).
We are running Cassandra 2.0.4.
We are running on a virtual machine with Xen.
We have 16GB of ram and default memory settings for C* (i.e. heap size of
4GB). CPU specified as 8 cores by our provider.

Right now, we have no data on the machine and no requests to it at all.
Still we get ParNew GCs like the following:
INFO [ScheduledTasks:1] 2014-01-18 10:54:42,286 GCInspector.java (line 116)
GC for ParNew: 464 ms for 1 collections, 102838776 used; max is 4106223616

While this may not be extremely long, on other machines with the same setup
but some data (around 12GB) and around 10 read requests/s (i.e. basically
no load) we have seen ParNew GC for 20 minutes or more. During this time,
the machine goes down completely (I can't even ssh to it). The requests are
mostly from OpsCenter and the rows requested are not extremely large
(typically less than 1KB).

We have tried a lot of different things to solve these issues since we've
been having them for a long time including:
- Upgrading Cassandra to new versions
- Upgrading Java to new versions
- Printing promotion failures in GC-log (no failures found!)
- Different sizes of heap and heap space for different GC spaces (Eden etc.)
- Different versions of Ubuntu
- Running on Amazon EC2 instead of the provider we are using now (not with
Datastax AMI)

Something that may be a clue is that when running the DataStax Community
AMI on Amazon we haven't seen the GC yet (it's been running for a week or
so). Just to be clear, another test on Amazon EC2 mentioned above (without
the Datastax AMI) shows the GC freezes.

If any other information is needed, just let me know.

Best regards,
Joel Samuelsson

Mime
View raw message