cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thakrar, Jayesh" <jthak...@conversantmedia.com>
Subject Any way to control/limit off-heap memory?
Date Sat, 04 Mar 2017 20:54:05 GMT
I have a situation where the off-heap memory is bloating the jvm process memory, making it
a candidate to be killed by the oom_killer.
My server has 256 GB RAM and Cassandra heap memory of 16 GB

Below is the output of "nodetool info" and nodetool compactionstats for a culprit table which
causes bloom filter bloat.
Ofcourse one option is to turnoff bloom filter, but I need to look into application access
pattern, etc.


xss =  -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError
-Xss256k
ID                     : 2b9b4252-0760-49c1-8d14-544be0183271
Gossip active          : true
Thrift active          : false
Native Transport active: true
Load                   : 953.19 GB
Generation No          : 1488641545
Uptime (seconds)       : 15706
Heap Memory (MB)       : 7692.93 / 16309.00
Off Heap Memory (MB)   : 175115.07
Data Center            : ord
Rack                   : rack3
Exceptions             : 0
Key Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN
recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN
recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN
recent hit rate, 7200 save period in seconds
Token                  : (invoke with -T/--tokens to see all 256 tokens)


                Table: logs_by_user
                SSTable count: 622
                SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
                Space used (live): 313156769247
                Space used (total): 313156769247
                Space used by snapshots (total): 0
                Off heap memory used (total): 180354511884
                SSTable Compression Ratio: 0.25016314078395613
                Number of keys (estimate): 147261312
                Memtable cell count: 44796
                Memtable data size: 57578717
                Memtable off heap memory used: 0
                Memtable switch count: 21
                Local read count: 0
                Local read latency: NaN ms
                Local write count: 1148687
                Local write latency: 0.123 ms
                Pending flushes: 0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 180269125192
                Bloom filter off heap memory used: 180269120216
                Index summary off heap memory used: 24335340
                Compression metadata off heap memory used: 61056328
                Compacted partition minimum bytes: 150
                Compacted partition maximum bytes: 668489532
                Compacted partition mean bytes: 3539
                Average live cells per slice (last five minutes): NaN
                Maximum live cells per slice (last five minutes): 0
                Average tombstones per slice (last five minutes): NaN
                Maximum tombstones per slice (last five minutes): 0


From: Conversant <jthak<mailto:jthakrar@conversantmedia.com>
Mime
View raw message