cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Lohfink <clohfin...@gmail.com>
Subject Re: gc causes C* node hang
Date Thu, 23 Nov 2017 20:46:15 GMT
Can you include output from the gc logs on the 30ms pause? If you dont have
gclogs, enable it and collect one. G1 provides good details and can catch
some edge cases with usecase.

I would guess since its so long you didnt have enough to-space. can try
adding -XX:G1ReservePercent=25 (or -XX:-G1UseAdaptiveIHOP and
-XX:InitiatingHeapOccupancyPercent) and increasing heap space if you can.

JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=5"
> JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=5"


how many cpu cores do you have? Make sure your not setting these lower than
default. ( check with `java -XX:+PrintFlagsFinal 2>&1 | grep Threads`)

Looks like 16gb heap?  how much space is available on the host (how big can
you set it)? swap disabled?

If its not to-space exhausted issue, gc logs will help.

Chris

On Thu, Nov 23, 2017 at 12:49 AM, Peng Xiao <2535053@qq.com> wrote:

> Hi there,
>
> We have a cluster with two DCs with 2.1.13,sometimes the gc will cause one
> node hang,and the application rt will jump to 15s,actually even we have one
> node down,the rt will not fluctuates violently.
> We are using Cassandra G1 with the following configuration:
>
> JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=5"
> JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=5"
> JVM_OPTS="$JVM_OPTS -XX:+ParallelRefProcEnabled"
>
> JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
> JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500"
> JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=5"
> JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25"
>
> Could anyone please advise?
>
>
> Thanks,
> Peng Xiao
>

Mime
View raw message