incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor <i...@4friends.od.ua>
Subject Re: Reduce Cassandra GC
Date Fri, 07 Jun 2013 19:42:50 GMT
If you are talking about 1.2.x then I also have memory problems on the 
idle cluster: java memory constantly slow grows up to limit, then spend 
long time for GC. I never seen such behaviour for 1.0.x and 1.1.x, where 
on idle cluster java memory stay on the same value.

On 06/07/2013 05:19 PM, Joel Samuelsson wrote:
> I keep having issues with GC. Besides the cluster mentioned above, we 
> also have a single node development cluster having the same issues. 
> This node has 12.33 GB data, a couple of million skinny rows and 
> basically no load. It has default memory settings but keep getting 
> very long stop-the-world GC pauses:
> INFO [ScheduledTasks:1] 2013-06-07 10:37:02,537 GCInspector.java (line 
> 122) GC for ParNew: 99342 ms for 1 collections, 1400754488 used; max 
> is 4114612224
> To try to rule out amount of memory, I set it to 16GB (we're on a 
> virtual environment), with 4GB of it for Cassandra heap but that 
> didn't help either, the incredibly long GC pauses keep coming.
> So I think something else is causing these issues, unless everyone is 
> having really long GC pauses (which I doubt). I came across this thread:
> http://www.mail-archive.com/user@cassandra.apache.org/msg24042.html
> suggesting # date -s “`date`” might help my issues. It didn't however 
> (unless I am supposed to replace that second date with the actual date?).
>
> Has anyone had similar issues?
>
>
> 2013/4/17 aaron morton <aaron@thelastpickle.com 
> <mailto:aaron@thelastpickle.com>>
>
>     > INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java
>     (line 122) GC for ParNew: 338798 ms for 1 collections, 592212416
>     used; max is 1046937600
>     This does not say that the heap is full.
>     ParNew is GC activity for the new heap, which is typically a
>     smaller part of the overall heap.
>
>     It sounds like you are running with defaults for the memory
>     config, which is generally a good idea. But 4GB total memory for a
>     node is on the small size.
>
>     Try some changes, edit the cassandra-env.sh file and change
>
>     MAX_HEAP_SIZE="2G"
>     HEAP_NEWSIZE="400M"
>
>     You may also want to try:
>
>     MAX_HEAP_SIZE="2G"
>     HEAP_NEWSIZE="800M"
>     JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=4"
>     JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=2"
>
>     The size of the new heap generally depends on the number of cores
>     available, see the commends in the -env file.
>
>     An older discussion about memory use, not that in 1.2 the bloom
>     filters (and compression data) are off heap now.
>     http://www.mail-archive.com/user@cassandra.apache.org/msg25762.html
>
>     Hope that helps.
>
>     -----------------
>     Aaron Morton
>     Freelance Cassandra Consultant
>     New Zealand
>
>     @aaronmorton
>     http://www.thelastpickle.com
>
>     On 17/04/2013, at 11:06 PM, Joel Samuelsson
>     <samuelsson.joel@gmail.com <mailto:samuelsson.joel@gmail.com>> wrote:
>
>     > You're right, it's probably hard. I should have provided more data.
>     >
>     > I'm running Ubuntu 10.04 LTS with JNA installed. I believe this
>     line in the log indicates that JNA is working, please correct me
>     if I'm wrong:
>     > CLibrary.java (line 111) JNA mlockall successful
>     >
>     > Total amount of RAM is 4GB.
>     >
>     > My description of data size was very bad. Sorry about that. Data
>     set size is 12.3 GB per node, compressed.
>     >
>     > Heap size is 998.44MB according to nodetool info.
>     > Key cache is 49MB bytes according to nodetool info.
>     > Row cache size is 0 bytes acoording to nodetool info.
>     > Max new heap is 205MB kbytes according to Memory Pool "Par Eden
>     Space" max in jconsole.
>     > Memtable is left at default which should give it 333MB according
>     to documentation (uncertain where I can verify this).
>     >
>     > Our production cluster seems similar to your dev cluster so
>     possibly increasing the heap to 2GB might help our issues.
>     >
>     > I am still interested in getting rough estimates of how much
>     heap will be needed as data grows. Other than empirical studies
>     how would I go about getting such estimates?
>     >
>     >
>     > 2013/4/16 Viktor Jevdokimov <Viktor.Jevdokimov@adform.com
>     <mailto:Viktor.Jevdokimov@adform.com>>
>     > How one could provide any help without any knowledge about your
>     cluster, node and environment settings?
>     >
>     >
>     >
>     > 40GB was calculated from 2 nodes with RF=2 (each has 100% data
>     range), 2.4-2.5M rows * 6 cols * 3kB as a minimum without
>     compression and any overhead (sstable, bloom filters and indexes).
>     >
>     >
>     >
>     > With ParNew GC time such as yours even if it is a swapping issue
>     I could say only that heap size is too small.
>     >
>     >
>     >
>     > Check Heap, New Heap sizes, memtable and cache sizes. Are you on
>     Linux? Is JNA installed and used? What is total amount of RAM?
>     >
>     >
>     >
>     > Just for a DEV environment we use 3 virtual machines with 4GB
>     RAM and use 2GB heap without any GC issue with amount of data from
>     0 to 16GB compressed on each node. Memtable space sized to 100MB,
>     New Heap 400MB.
>     >
>     >
>     >
>     > Best regards / Pagarbiai
>     > Viktor Jevdokimov
>     > Senior Developer
>     >
>     > Email: Viktor.Jevdokimov@adform.com
>     <mailto:Viktor.Jevdokimov@adform.com>
>     > Phone: +370 5 212 3063 <tel:%2B370%205%20212%203063>, Fax +370 5
>     261 0453 <tel:%2B370%205%20261%200453>
>     > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
>     > Follow us on Twitter: @adforminsider
>     > Take a ride with Adform's Rich Media Suite
>     > <signature-logo29.png>
>     > <signature-best-employer-logo4823.png>
>     >
>     > Disclaimer: The information contained in this message and
>     attachments is intended solely for the attention and use of the
>     named addressee and may be confidential. If you are not the
>     intended recipient, you are reminded that the information remains
>     the property of the sender. You must not use, disclose,
>     distribute, copy, print or rely on this e-mail. If you have
>     received this message in error, please contact the sender
>     immediately and irrevocably delete this message and any copies.
>     >
>     > From: Joel Samuelsson [mailto:samuelsson.joel@gmail.com
>     <mailto:samuelsson.joel@gmail.com>]
>     > Sent: Tuesday, April 16, 2013 12:52
>     > To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
>     > Subject: Re: Reduce Cassandra GC
>     >
>     >
>     >
>     > How do you calculate the heap / data size ratio? Is this a
>     linear ratio?
>     >
>     >
>     >
>     > Each node has slightly more than 12 GB right now though.
>     >
>     >
>     >
>     > 2013/4/16 Viktor Jevdokimov <Viktor.Jevdokimov@adform.com
>     <mailto:Viktor.Jevdokimov@adform.com>>
>     >
>     > For a >40GB of data 1GB of heap is too low.
>     >
>     >
>     >
>     > Best regards / Pagarbiai
>     >
>     > Viktor Jevdokimov
>     >
>     > Senior Developer
>     >
>     >
>     >
>     > Email: Viktor.Jevdokimov@adform.com
>     <mailto:Viktor.Jevdokimov@adform.com>
>     >
>     > Phone: +370 5 212 3063 <tel:%2B370%205%20212%203063>, Fax +370 5
>     261 0453 <tel:%2B370%205%20261%200453>
>     >
>     > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
>     >
>     > Follow us on Twitter: @adforminsider
>     >
>     > Take a ride with Adform's Rich Media Suite
>     >
>     > <image001.png>
>     >
>     > <image002.png>
>     >
>     >
>     > Disclaimer: The information contained in this message and
>     attachments is intended solely for the attention and use of the
>     named addressee and may be confidential. If you are not the
>     intended recipient, you are reminded that the information remains
>     the property of the sender. You must not use, disclose,
>     distribute, copy, print or rely on this e-mail. If you have
>     received this message in error, please contact the sender
>     immediately and irrevocably delete this message and any copies.
>     >
>     >
>     >
>     > From: Joel Samuelsson [mailto:samuelsson.joel@gmail.com
>     <mailto:samuelsson.joel@gmail.com>]
>     > Sent: Tuesday, April 16, 2013 10:47
>     > To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
>     > Subject: Reduce Cassandra GC
>     >
>     >
>     >
>     > Hi,
>     >
>     >
>     >
>     > We have a small production cluster with two nodes. The load on
>     the nodes is very small, around 20 reads / sec and about the same
>     for writes. There are around 2.5 million keys in the cluster and a
>     RF of 2.
>     >
>     >
>     >
>     > About 2.4 million of the rows are skinny (6 columns) and around
>     3kb in size (each). Currently, scripts are running, accessing all
>     of the keys in timeorder to do some calculations.
>     >
>     >
>     >
>     > While running the scripts, the nodes go down and then come back
>     up 6-7 minutes later. This seems to be due to GC. I get lines like
>     this in the log:
>     >
>     > INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java
>     (line 122) GC for ParNew: 338798 ms for 1 collections, 592212416
>     used; max is 1046937600
>     >
>     >
>     >
>     > However, the heap is not full. The heap usage has a jagged
>     pattern going from 60% up to 70% during 5 minutes and then back
>     down to 60% the next 5 minutes and so on. I get no "Heap is X
>     full..." messages. Every once in a while at one of these peaks, I
>     get these stop-the-world GC for 6-7 minutes. Why does GC take up
>     so much time even though the heap isn't full?
>     >
>     >
>     >
>     > I am aware that my access patterns make key caching very
>     unlikely to be high. And indeed, my average key cache hit ratio
>     during the run of the scripts is around 0.5%. I tried disabling
>     key caching on the accessed column family (UPDATE COLUMN FAMILY cf
>     WITH caching=none;) through the cassandra-cli but I get the same
>     behaviour. Is the turning key cache off effective immediately?
>     >
>     >
>     >
>     > Stop-the-world GC is fine if it happens for a few seconds but
>     having them for several minutes doesn't work. Any other
>     suggestions to remove them?
>     >
>     >
>     >
>     > Best regards,
>     >
>     > Joel Samuelsson
>     >
>     >
>     >
>     >
>
>


Mime
View raw message