incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra services down frequently [Version 1.1.4]
Date Tue, 09 Apr 2013 07:52:36 GMT
> MAX_HEAP_SIZE="6G"
> HEAP_NEWSIZE="500M"
The new heap feels a little low, I often see 800M as a good number. It depends on the number
of cores, but if that's working stick with it. 

> key_cache_size_in_mb: 512
Have you run this at the default and checked the cache hit rate using nodetool info ? The
default size would be about 300M. 

> row_cache_size_in_mb: 14336
This is way too high. 
You've told the JVM to lock in 6GB and then told the row cache it can use 14GB, but you only
have 16GB on the node. At some point things are going to go crash, bang, wallop. 

Set it to 1GB and check the cache hit rate using nodetool info. 

The remaining memory will be used by the OS to cache disk access. 


> I have a querry, if Cassandra is using JVM for all operations then why we need change
above parameters separately in cassandra.yaml.

The JVM params are passed to the JVM before the server starts and have to be formatted a specific
way. The yaml file is much easier for humans to read. 

Cheers


-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 8/04/2013, at 1:16 PM, 金剑 <jinjian.1@gmail.com> wrote:

> It also use off-heap memory out of JVM. SerializingCacheProvider should be one of the
case.
> 
> Best Regards!
> 
> Jian Jin
> 
> 
> 2013/4/6 <adeel.akbar@panasiangroup.com>
> Thank you Aaron and Bryan for your advice.
> 
> I have changed following parameters and now Cassandra running absolutely fine. Please
review below setting and advice am I right or right direction.
> 
> cassandra-env.sh
> #JVM_OPTS="$JVM_OPTS -ea"
> MAX_HEAP_SIZE="6G"
> HEAP_NEWSIZE="500M"
> 
>  cassandra.yaml
> # do not persist caches to disk
> key_cache_save_period: 0
> row_cache_save_period: 0
> 
> key_cache_size_in_mb: 512
> row_cache_size_in_mb: 14336
> row_cache_provider: SerializingCacheProvider
> 
> I have a querry, if Cassandra is using JVM for all operations then why we need change
above parameters separately in cassandra.yaml.
> 
> 
> Thanks & Regards
> 
> Adeel Akbar
> 
> 
> Quoting aaron morton <aaron@thelastpickle.com>:
> 
> We can see from below that you've tweaked and disabled many of the  memory "safety valve"
and other memory related settings.
> Agree.
> Also you are running with JVM heap size of 3.81GB which is non  default. For a 16GB node
I would expect 8GB.
> 
> Try restoring the yaml values to the defaults and allowing the  cassandra-env.sh file
to determine the memory size.
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 5/04/2013, at 12:36 PM, Bryan Talbot <btalbot@aeriagames.com> wrote:
> 
> On Thu, Apr 4, 2013 at 1:27 AM, <adeel.akbar@panasiangroup.com> wrote:
> 
> After some time (1 hour / 2 hour) cassandra shut services on one or  two nodes with follwoing
errors;
> 
> 
> Wonder what the workload and schema is like ...
> 
> We can see from below that you've tweaked and disabled many of the  memory "safety valve"
and other memory related settings.  Those  could be causing issues too.
> 
> 
> hinted_handoff_throttle_delay_in_ms: 0
> flush_largest_memtables_at: 1.0
> reduce_cache_sizes_at: 1.0
> reduce_cache_capacity_to: 0.6
> rpc_keepalive: true
> rpc_server_type: sync
> rpc_min_threads: 16
> rpc_max_threads: 2147483647
> in_memory_compaction_limit_in_mb: 256
> compaction_throughput_mb_per_sec: 16
> rpc_timeout_in_ms: 15000
> dynamic_snitch_badness_threshold: 0.0
> 
> 
> 
> 


Mime
View raw message