It also use off-heap memory out of JVM. SerializingCacheProvider should be one of the case.

Best Regards!

Jian Jin



2013/4/6 <adeel.akbar@panasiangroup.com>
Thank you Aaron and Bryan for your advice.

I have changed following parameters and now Cassandra running absolutely fine. Please review below setting and advice am I right or right direction.

cassandra-env.sh
#JVM_OPTS="$JVM_OPTS -ea"
MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="500M"

 cassandra.yaml
# do not persist caches to disk
key_cache_save_period: 0
row_cache_save_period: 0

key_cache_size_in_mb: 512
row_cache_size_in_mb: 14336
row_cache_provider: SerializingCacheProvider

I have a querry, if Cassandra is using JVM for all operations then why we need change above parameters separately in cassandra.yaml.


Thanks & Regards

Adeel Akbar


Quoting aaron morton <aaron@thelastpickle.com>:

We can see from below that you've tweaked and disabled many of the  memory "safety valve" and other memory related settings.
Agree.
Also you are running with JVM heap size of 3.81GB which is non  default. For a 16GB node I would expect 8GB.

Try restoring the yaml values to the defaults and allowing the  cassandra-env.sh file to determine the memory size.

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 5/04/2013, at 12:36 PM, Bryan Talbot <btalbot@aeriagames.com> wrote:

On Thu, Apr 4, 2013 at 1:27 AM, <adeel.akbar@panasiangroup.com> wrote:

After some time (1 hour / 2 hour) cassandra shut services on one or  two nodes with follwoing errors;


Wonder what the workload and schema is like ...

We can see from below that you've tweaked and disabled many of the  memory "safety valve" and other memory related settings.  Those  could be causing issues too.


hinted_handoff_throttle_delay_in_ms: 0
flush_largest_memtables_at: 1.0
reduce_cache_sizes_at: 1.0
reduce_cache_capacity_to: 0.6
rpc_keepalive: true
rpc_server_type: sync
rpc_min_threads: 16
rpc_max_threads: 2147483647
in_memory_compaction_limit_in_mb: 256
compaction_throughput_mb_per_sec: 16
rpc_timeout_in_ms: 15000
dynamic_snitch_badness_threshold: 0.0