cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-2543) Node not responding, bringing down cluster, marked as up
Date Fri, 22 Apr 2011 14:58:05 GMT


Jonathan Ellis commented on CASSANDRA-2543:

bq. It still interest me why the node didn't recover even though there were nearly no requests
at all from our application

the two big users of memory are usually memtables and cache and neither one gets freed when
requests stop. (until memtable_flush_after_mins is reached...  and flush needs memory to work
so if you are GC storming hard enough it may not make progress even then.)

> Node not responding, bringing down cluster, marked as up
> --------------------------------------------------------
>                 Key: CASSANDRA-2543
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.4
>            Reporter: Thibaut
>             Fix For: 0.7.6
>         Attachments: jstack
> I have one node which constantly hangs and brings done the entire cluster (not giving
any answers).
> If I restart the node, the node will hang after a certain number of time. I have no indication
> It's marked as up when executing the nodetool ring command.
> Executing the ring command on the node itself (without any traffic on the cluster) takes
at least 2 minutes to execute. The node takes about 50%-100% of cpu over all cpus.
> Netstats doesn't list anything interesting:
> /software/cassandra/bin/nodetool -h localhost netstats
> Mode: Normal
> Not sending any streams.
> Not receiving any streams.
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0          51064
> Responses                       n/a         0         530479
> I attached the jstack of the node. There are no indications that the node has faulty
> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms5254M -Xmx5254M
-Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Dlog4j.defaultInitOverride=true -Dcassandra-foreground=yes
-cp /software/cassandra/bin/../conf:/software/cassandra/bin/../build/classes:/software/cassandra/bin/../lib/antlr-3.1.3.jar:/software/cassandra/bin/../lib/apache-cassandra-0.7.4.jar:/software/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/software/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/software/cassandra/bin/../lib/commons-cli-1.1.jar:/software/cassandra/bin/../lib/commons-codec-1.2.jar:/software/cassandra/bin/../lib/commons-collections-3.2.1.jar:/software/cassandra/bin/../lib/commons-lang-2.4.jar:/software/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.1.jar:/software/cassandra/bin/../lib/guava-r05.jar:/software/cassandra/bin/../lib/high-scale-lib.jar:/software/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/software/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/software/cassandra/bin/../lib/jetty-6.1.21.jar:/software/cassandra/bin/../lib/jetty-util-6.1.21.jar:/software/cassandra/bin/../lib/jline-0.9.94.jar:/software/cassandra/bin/../lib/json-simple-1.1.jar:/software/cassandra/bin/../lib/jug-2.0.0.jar:/software/cassandra/bin/../lib/libthrift-0.5.jar:/software/cassandra/bin/../lib/log4j-1.2.16.jar:/software/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/software/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/software/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/software/cassandra/bin/../lib/snakeyaml-1.6.jar

This message is automatically generated by JIRA.
For more information on JIRA, see:

View raw message