incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Samuelsson <samuelsson.j...@gmail.com>
Subject Re: Reduce Cassandra GC
Date Fri, 07 Jun 2013 14:19:53 GMT
I keep having issues with GC. Besides the cluster mentioned above, we also
have a single node development cluster having the same issues. This node
has 12.33 GB data, a couple of million skinny rows and basically no load.
It has default memory settings but keep getting very long stop-the-world GC
pauses:
INFO [ScheduledTasks:1] 2013-06-07 10:37:02,537 GCInspector.java (line 122)
GC for ParNew: 99342 ms for 1 collections, 1400754488 used; max is
4114612224
To try to rule out amount of memory, I set it to 16GB (we're on a virtual
environment), with 4GB of it for Cassandra heap but that didn't help
either, the incredibly long GC pauses keep coming.
So I think something else is causing these issues, unless everyone is
having really long GC pauses (which I doubt). I came across this thread:
http://www.mail-archive.com/user@cassandra.apache.org/msg24042.html
suggesting # date -s “`date`” might help my issues. It didn't however
(unless I am supposed to replace that second date with the actual date?).

Has anyone had similar issues?


2013/4/17 aaron morton <aaron@thelastpickle.com>

> > INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line
> 122) GC for ParNew: 338798 ms for 1 collections, 592212416 used; max is
> 1046937600
> This does not say that the heap is full.
> ParNew is GC activity for the new heap, which is typically a smaller part
> of the overall heap.
>
> It sounds like you are running with defaults for the memory config, which
> is generally a good idea. But 4GB total memory for a node is on the small
> size.
>
> Try some changes, edit the cassandra-env.sh file and change
>
> MAX_HEAP_SIZE="2G"
> HEAP_NEWSIZE="400M"
>
> You may also want to try:
>
> MAX_HEAP_SIZE="2G"
> HEAP_NEWSIZE="800M"
> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=4"
> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=2"
>
> The size of the new heap generally depends on the number of cores
> available, see the commends in the -env file.
>
> An older discussion about memory use, not that in 1.2 the bloom filters
> (and compression data) are off heap now.
> http://www.mail-archive.com/user@cassandra.apache.org/msg25762.html
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 17/04/2013, at 11:06 PM, Joel Samuelsson <samuelsson.joel@gmail.com>
> wrote:
>
> > You're right, it's probably hard. I should have provided more data.
> >
> > I'm running Ubuntu 10.04 LTS with JNA installed. I believe this line in
> the log indicates that JNA is working, please correct me if I'm wrong:
> > CLibrary.java (line 111) JNA mlockall successful
> >
> > Total amount of RAM is 4GB.
> >
> > My description of data size was very bad. Sorry about that. Data set
> size is 12.3 GB per node, compressed.
> >
> > Heap size is 998.44MB according to nodetool info.
> > Key cache is 49MB bytes according to nodetool info.
> > Row cache size is 0 bytes acoording to nodetool info.
> > Max new heap is 205MB kbytes according to Memory Pool "Par Eden Space"
> max in jconsole.
> > Memtable is left at default which should give it 333MB according to
> documentation (uncertain where I can verify this).
> >
> > Our production cluster seems similar to your dev cluster so possibly
> increasing the heap to 2GB might help our issues.
> >
> > I am still interested in getting rough estimates of how much heap will
> be needed as data grows. Other than empirical studies how would I go about
> getting such estimates?
> >
> >
> > 2013/4/16 Viktor Jevdokimov <Viktor.Jevdokimov@adform.com>
> > How one could provide any help without any knowledge about your cluster,
> node and environment settings?
> >
> >
> >
> > 40GB was calculated from 2 nodes with RF=2 (each has 100% data range),
> 2.4-2.5M rows * 6 cols * 3kB as a minimum without compression and any
> overhead (sstable, bloom filters and indexes).
> >
> >
> >
> > With ParNew GC time such as yours even if it is a swapping issue I could
> say only that heap size is too small.
> >
> >
> >
> > Check Heap, New Heap sizes, memtable and cache sizes. Are you on Linux?
> Is JNA installed and used? What is total amount of RAM?
> >
> >
> >
> > Just for a DEV environment we use 3 virtual machines with 4GB RAM and
> use 2GB heap without any GC issue with amount of data from 0 to 16GB
> compressed on each node. Memtable space sized to 100MB, New Heap 400MB.
> >
> >
> >
> > Best regards / Pagarbiai
> > Viktor Jevdokimov
> > Senior Developer
> >
> > Email: Viktor.Jevdokimov@adform.com
> > Phone: +370 5 212 3063, Fax +370 5 261 0453
> > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> > Follow us on Twitter: @adforminsider
> > Take a ride with Adform's Rich Media Suite
> > <signature-logo29.png>
> > <signature-best-employer-logo4823.png>
> >
> > Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
> >
> > From: Joel Samuelsson [mailto:samuelsson.joel@gmail.com]
> > Sent: Tuesday, April 16, 2013 12:52
> > To: user@cassandra.apache.org
> > Subject: Re: Reduce Cassandra GC
> >
> >
> >
> > How do you calculate the heap / data size ratio? Is this a linear ratio?
> >
> >
> >
> > Each node has slightly more than 12 GB right now though.
> >
> >
> >
> > 2013/4/16 Viktor Jevdokimov <Viktor.Jevdokimov@adform.com>
> >
> > For a >40GB of data 1GB of heap is too low.
> >
> >
> >
> > Best regards / Pagarbiai
> >
> > Viktor Jevdokimov
> >
> > Senior Developer
> >
> >
> >
> > Email: Viktor.Jevdokimov@adform.com
> >
> > Phone: +370 5 212 3063, Fax +370 5 261 0453
> >
> > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> >
> > Follow us on Twitter: @adforminsider
> >
> > Take a ride with Adform's Rich Media Suite
> >
> > <image001.png>
> >
> > <image002.png>
> >
> >
> > Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
> >
> >
> >
> > From: Joel Samuelsson [mailto:samuelsson.joel@gmail.com]
> > Sent: Tuesday, April 16, 2013 10:47
> > To: user@cassandra.apache.org
> > Subject: Reduce Cassandra GC
> >
> >
> >
> > Hi,
> >
> >
> >
> > We have a small production cluster with two nodes. The load on the nodes
> is very small, around 20 reads / sec and about the same for writes. There
> are around 2.5 million keys in the cluster and a RF of 2.
> >
> >
> >
> > About 2.4 million of the rows are skinny (6 columns) and around 3kb in
> size (each). Currently, scripts are running, accessing all of the keys in
> timeorder to do some calculations.
> >
> >
> >
> > While running the scripts, the nodes go down and then come back up 6-7
> minutes later. This seems to be due to GC. I get lines like this in the log:
> >
> > INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line
> 122) GC for ParNew: 338798 ms for 1 collections, 592212416 used; max is
> 1046937600
> >
> >
> >
> > However, the heap is not full. The heap usage has a jagged pattern going
> from 60% up to 70% during 5 minutes and then back down to 60% the next 5
> minutes and so on. I get no "Heap is X full..." messages. Every once in a
> while at one of these peaks, I get these stop-the-world GC for 6-7 minutes.
> Why does GC take up so much time even though the heap isn't full?
> >
> >
> >
> > I am aware that my access patterns make key caching very unlikely to be
> high. And indeed, my average key cache hit ratio during the run of the
> scripts is around 0.5%. I tried disabling key caching on the accessed
> column family (UPDATE COLUMN FAMILY cf WITH caching=none;) through the
> cassandra-cli but I get the same behaviour. Is the turning key cache off
> effective immediately?
> >
> >
> >
> > Stop-the-world GC is fine if it happens for a few seconds but having
> them for several minutes doesn't work. Any other suggestions to remove them?
> >
> >
> >
> > Best regards,
> >
> > Joel Samuelsson
> >
> >
> >
> >
>
>

Mime
View raw message