incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chip Salzenberg <rev.c...@gmail.com>
Subject Gossip yoyo under write load
Date Fri, 12 Nov 2010 21:19:21 GMT
After I rebooted my 0.7.0beta3+ cluster to increase threads (read=100
write=200 ... they're beefy machines), and putting them under load again, I
find gossip reporting yoyo up-down-up-down status for the other nodes.
 Anyone know what this is a symptom of, and/or how to avoid it?  I haven't
seen any particular symptoms other than the log messages; and I suppose I'm
also dropping replication MUTATEs which had been happening already, anyway.

cas001   INFO [ScheduledTasks:1] 2010-11-12 13:00:02,891 Gossiper.java (line
195) InetAddress /X.20 is now dead.
cas001   INFO [GossipStage:1] 2010-11-12 13:00:07,567 Gossiper.java (line
569) InetAddress /X.20 is now UP
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:00:53,662 Gossiper.java (line
195) InetAddress /X.21 is now dead.
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:00:56,967 GCInspector.java
(line 133) GC for ParNew: 255 ms, 135668944 reclaimed leaving 18375966648
used; max is 34557919232
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:01:01,135
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289595661135.log
cas001   INFO [GossipStage:1] 2010-11-12 13:01:08,148 Gossiper.java (line
569) InetAddress /X.21 is now UP
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:01:56,753 GCInspector.java
(line 133) GC for ParNew: 268 ms, 132609096 reclaimed leaving 20102566032
used; max is 34557919232
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:01:57,771 GCInspector.java
(line 133) GC for ParNew: 274 ms, 115223104 reclaimed leaving 20214228560
used; max is 34557919232
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:02:14,746
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289595734746.log
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:03:02,868 GCInspector.java
(line 133) GC for ParNew: 297 ms, 62163960 reclaimed leaving 22200082216
used; max is 34557919232
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:03:29,123
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289595809123.log
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:04:09,626 GCInspector.java
(line 133) GC for ParNew: 321 ms, 125585880 reclaimed leaving 24138058936
used; max is 34557919232
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:04:44,852
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289595884852.log
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:05:13,695 GCInspector.java
(line 133) GC for ParNew: 242 ms, 126754312 reclaimed leaving 26019407576
used; max is 34557919232
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:06:01,941
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289595961941.log
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:06:25,617 GCInspector.java
(line 133) GC for ParNew: 307 ms, 134631824 reclaimed leaving 11283839952
used; max is 34557919232
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:06:37,032 Gossiper.java (line
195) InetAddress /X.18 is now dead.
cas001   INFO [GossipStage:1] 2010-11-12 13:06:38,666 Gossiper.java (line
569) InetAddress /X.18 is now UP
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:07:23,417
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289596043417.log
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:07:33,034 GCInspector.java
(line 133) GC for ParNew: 231 ms, 108391848 reclaimed leaving 13146098816
used; max is 34557919232
cas001   INFO [MutationStage:169] 2010-11-12 13:08:12,548
ColumnFamilyStore.java (line 580) switching in a fresh Memtable for
TestAttrs at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1289596043417.log',
position=84947614)
cas001   INFO [MutationStage:169] 2010-11-12 13:08:12,549
ColumnFamilyStore.java (line 879) Enqueuing flush of
Memtable-TestAttrs@1721243764(293461476 bytes, 8388614 operations)
cas001   INFO [FlushWriter:1] 2010-11-12 13:08:12,549 Memtable.java (line
155) Writing Memtable-TestAttrs@1721243764(293461476 bytes, 8388614
operations)
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:08:40,628 GCInspector.java
(line 133) GC for ParNew: 278 ms, 135521080 reclaimed leaving 15121172544
used; max is 34557919232
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:08:42,349
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289596122349.log
cas001   INFO [FlushWriter:1] 2010-11-12 13:09:20,586 Memtable.java (line
162) Completed flushing
/var/lib/cassandra/data/Attrs/TestAttrs-e-305-Data.db (954244537 bytes)
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:09:31,873 GCInspector.java
(line 133) GC for ParNew: 228 ms, 201874960 reclaimed leaving 16512318064
used; max is 34557919232
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:09:47,127 GCInspector.java
(line 133) GC for ParNew: 266 ms, 123236640 reclaimed leaving 17009227136
used; max is 34557919232
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:10:01,895
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289596201895.log
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:10:48,377 GCInspector.java
(line 133) GC for ParNew: 296 ms, 124419344 reclaimed leaving 18796530912
used; max is 34557919232
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:10:54,391 Gossiper.java (line
195) InetAddress /X.21 is now dead.
cas001   INFO [GossipStage:1] 2010-11-12 13:10:57,688 Gossiper.java (line
569) InetAddress /X.21 is now UP
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:11:17,968
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289596277968.log
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:11:47,678 GCInspector.java
(line 133) GC for ParNew: 281 ms, 128569912 reclaimed leaving 20548892048
used; max is 34557919232
cas001   INFO [COMMIT-LOG-WRITER] 2010-11-12 13:12:33,745
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1289596353745.log
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:12:59,242 GCInspector.java
(line 133) GC for ParNew: 253 ms, 109047624 reclaimed leaving 22601534128
used; max is 34557919232
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:13:30,583 GCInspector.java
(line 133) GC for ParNew: 201 ms, 202760624 reclaimed leaving 23439605592
used; max is 34557919232
cas001   INFO [ScheduledTasks:1] 2010-11-12 13:13:32,586 Gossiper.java (line
195) InetAddress /X.14 is now dead.
cas001   INFO [GossipStage:1] 2010-11-12 13:13:33,979 Gossiper.java (line
569) InetAddress /X.14 is now UP

Mime
View raw message