cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peng Xiao" <2535...@qq.com>
Subject 回复: gc causes C* node hang
Date Thu, 23 Nov 2017 20:54:51 GMT
We only have 7 cores per node.
For XX:ParallelGCThreads ,looks By default, Hotspot caps GC threads at 8,
maybe we need to remove this?




------------------ 原始邮件 ------------------
发件人: "我自己的邮箱";<2535053@qq.com>;
发送时间: 2017年11月24日(星期五) 凌晨4:51
收件人: "user"<user@cassandra.apache.org>;

主题: 回复: gc causes C* node hang



Thanks Chris.we don't have gclogs for this node.we will try to add XX:G1ReservePercent=25.




------------------ 原始邮件 ------------------
发件人: "Chris Lohfink";<clohfink85@gmail.com>;
发送时间: 2017年11月24日(星期五) 凌晨4:46
收件人: "user"<user@cassandra.apache.org>;

主题: Re: gc causes C* node hang



Can you include output from the gc logs on the 30ms pause? If you dont have gclogs, enable
it and collect one. G1 provides good details and can catch some edge cases with usecase.

I would guess since its so long you didnt have enough to-space. can try adding -XX:G1ReservePercent=25
(or -XX:-G1UseAdaptiveIHOP and -XX:InitiatingHeapOccupancyPercent) and increasing heap space
if you can.

JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=5"
JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=5"

how many cpu cores do you have? Make sure your not setting these lower than default. ( check
with `java -XX:+PrintFlagsFinal 2>&1 | grep Threads`)
 

Looks like 16gb heap?  how much space is available on the host (how big can you set it)? swap
disabled?

If its not to-space exhausted issue, gc logs will help.

Chris






On Thu, Nov 23, 2017 at 12:49 AM, Peng Xiao <2535053@qq.com> wrote:
Hi there,


We have a cluster with two DCs with 2.1.13,sometimes the gc will cause one node hang,and the
application rt will jump to 15s,actually even we have one node down,the rt will not fluctuates
violently. 
We are using Cassandra G1 with the following configuration:


JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=5"
JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=5"
JVM_OPTS="$JVM_OPTS -XX:+ParallelRefProcEnabled"


JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500"
JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=5"
JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25"



Could anyone please advise?





Thanks,
Peng Xiao
Mime
View raw message