incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Healthy JVM GC
Date Tue, 12 Feb 2013 16:50:53 GMT
> -Xms8049M 
> -Xmx8049M 
> -Xmn800M 
That's a healthy amount of memory for the JVM. 

If you are using Row Caches, reduce their size and/or ensure you are using Serializing (off
heap) caches.
Also consider changing the yaml conf flush_largest_memtables_at from 0.75 to  0.80 so it is
different to the CMS occupancy setting. 
If you have a lot of rows, 100's of millions, consider reducing the bloom filter false positive
ratio. 

Or just upgrade to 1.2 which uses less JVM memory. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 9/02/2013, at 7:46 AM, André Cruz <andre.cruz@co.sapo.pt> wrote:

> Hello.
> 
> I've noticed I get the frequent JVM warning in the logs about the heap being full:
> 
> WARN [ScheduledTasks:1] 2013-02-08 18:14:20,410 GCInspector.java (line 145) Heap is 0.731554347747841
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> WARN [ScheduledTasks:1] 2013-02-08 18:14:20,418 StorageService.java (line 2855) Flushing
CFS(Keyspace='Disco', ColumnFamily='FilesPerBlock') to relieve memory pressure
> INFO [ScheduledTasks:1] 2013-02-08 18:14:20,418 ColumnFamilyStore.java (line 659) Enqueuing
flush of Memtable-FilesPerBlock@1804403938(6275300/63189158 serialized/live bytes, 52227 ops)
> INFO [FlushWriter:4500] 2013-02-08 18:14:20,419 Memtable.java (line 264) Writing Memtable-FilesPerBlock@1804403938(6275300/63189158
serialized/live bytes, 52227 ops)
> INFO [FlushWriter:4500] 2013-02-08 18:14:21,059 Memtable.java (line 305) Completed flushing
/servers/storage/cassandra-data/Disco/FilesPerBlock/Disco-FilesPerBlock-he-6154-Data.db (6332375
bytes) for commitlog position ReplayPosition(segmentId=1357730625412, position=10756636)
> WARN [ScheduledTasks:1] 2013-02-08 18:23:31,970 GCInspector.java (line 145) Heap is 0.6835904101057064
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> WARN [ScheduledTasks:1] 2013-02-08 18:23:31,971 StorageService.java (line 2855) Flushing
CFS(Keyspace='Disco', ColumnFamily='BlocksKnownPerUser') to relieve memory pressure
> INFO [ScheduledTasks:1] 2013-02-08 18:23:31,972 ColumnFamilyStore.java (line 659) Enqueuing
flush of Memtable-BlocksKnownPerUser@2072550435(1834642/60143054 serialized/live bytes, 67010
ops)
> INFO [FlushWriter:4501] 2013-02-08 18:23:31,972 Memtable.java (line 264) Writing Memtable-BlocksKnownPerUser@2072550435(1834642/60143054
serialized/live bytes, 67010 ops)
> INFO [FlushWriter:4501] 2013-02-08 18:23:32,827 Memtable.java (line 305) Completed flushing
/servers/storage/cassandra-data/Disco/BlocksKnownPerUser/Disco-BlocksKnownPerUser-he-484930-Data.db
(7404407 bytes) for commitlog position ReplayPosition(segmentId=1357730625413, position=6093472)
> WARN [ScheduledTasks:1] 2013-02-08 18:29:46,198 GCInspector.java (line 145) Heap is 0.6871977390878024
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> WARN [ScheduledTasks:1] 2013-02-08 18:29:46,199 StorageService.java (line 2855) Flushing
CFS(Keyspace='Disco', ColumnFamily='FileRevision') to relieve memory pressure
> INFO [ScheduledTasks:1] 2013-02-08 18:29:46,200 ColumnFamilyStore.java (line 659) Enqueuing
flush of Memtable-FileRevision@1526026442(7245147/63711465 serialized/live bytes, 23779 ops)
> INFO [FlushWriter:4502] 2013-02-08 18:29:46,201 Memtable.java (line 264) Writing Memtable-FileRevision@1526026442(7245147/63711465
serialized/live bytes, 23779 ops)
> INFO [FlushWriter:4502] 2013-02-08 18:29:46,769 Memtable.java (line 305) Completed flushing
/servers/storage/cassandra-data/Disco/FileRevision/Disco-FileRevision-he-5438-Data.db (5480642
bytes) for commitlog position ReplayPosition(segmentId=1357730625413, position=29816878)
> INFO [ScheduledTasks:1] 2013-02-08 18:34:13,442 GCInspector.java (line 122) GC for ConcurrentMarkSweep:
352 ms for 1 collections, 5902597760 used; max is 8357150720
> WARN [ScheduledTasks:1] 2013-02-08 18:34:13,442 GCInspector.java (line 145) Heap is 0.7062930845406603
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> WARN [ScheduledTasks:1] 2013-02-08 18:34:13,443 StorageService.java (line 2855) Flushing
CFS(Keyspace='Disco', ColumnFamily='NamespaceFile') to relieve memory pressure
> INFO [ScheduledTasks:1] 2013-02-08 18:34:13,443 ColumnFamilyStore.java (line 659) Enqueuing
flush of Memtable-NamespaceFile@1719163849(14395839/64692951 serialized/live bytes, 51765
ops)
> INFO [FlushWriter:4503] 2013-02-08 18:34:13,446 Memtable.java (line 264) Writing Memtable-NamespaceFile@1719163849(14395839/64692951
serialized/live bytes, 51765 ops)
> INFO [FlushWriter:4503] 2013-02-08 18:34:14,173 Memtable.java (line 305) Completed flushing
/servers/storage/cassandra-data/Disco/NamespaceFile/Disco-NamespaceFile-he-5516-Data.db (7747535
bytes) for commitlog position ReplayPosition(segmentId=1357730625414, position=14940209)
> WARN [ScheduledTasks:1] 2013-02-08 18:37:00,593 GCInspector.java (line 145) Heap is 0.6828223856659127
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> WARN [ScheduledTasks:1] 2013-02-08 18:37:00,593 StorageService.java (line 2855) Flushing
CFS(Keyspace='Disco', ColumnFamily='RevisionLog') to relieve memory pressure
> INFO [ScheduledTasks:1] 2013-02-08 18:37:00,594 ColumnFamilyStore.java (line 659) Enqueuing
flush of Memtable-RevisionLog@726433503(12434137/65978509 serialized/live bytes, 59638 ops)
> INFO [FlushWriter:4504] 2013-02-08 18:37:00,596 Memtable.java (line 264) Writing Memtable-RevisionLog@726433503(12434137/65978509
serialized/live bytes, 59638 ops)
> INFO [FlushWriter:4504] 2013-02-08 18:37:01,523 Memtable.java (line 305) Completed flushing
/servers/storage/cassandra-data/Disco/RevisionLog/Disco-RevisionLog-he-1664-Data.db (2366787
bytes) for commitlog position ReplayPosition(segmentId=1357730625414, position=27626400)
> 
> 
> Is this healthy GC behaviour? Or should I adjust the memory/gc settings? This machine
has 32GB RAM and my current JVM settings are:
> 
> -XX:+UseThreadPriorities 
> -XX:ThreadPriorityPolicy=42 
> -Xms8049M 
> -Xmx8049M 
> -Xmn800M 
> -XX:+HeapDumpOnOutOfMemoryError 
> -Xss196k 
> -XX:+UseParNewGC 
> -XX:+UseConcMarkSweepGC 
> -XX:+CMSParallelRemarkEnabled 
> -XX:SurvivorRatio=8 
> -XX:MaxTenuringThreshold=1 
> -XX:CMSInitiatingOccupancyFraction=75 
> -XX:+UseCMSInitiatingOccupancyOnly 
> 
> Thanks,
> André Cruz
> 
> 


Mime
View raw message