Assuming all the memory and yaml settings default that does not sound right.
The first thought would be the memory meter not counting correctly...
Do you do a lot of deletes ?
Do you have a lot of CF's and/or secondary indexes ?
Can you see log lines about the "liveRatio" for your cf's ?
I would upgrade to 1.0.10 before getting too carried away though.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 17/07/2012, at 8:14 PM, Nikolay Kоvshov wrote:
>
> This is a cluster of 2 nodes, each having 8G of operating memory, replicationfactor=2
> Write/read pressure is quite low and almost never exceeds 10/second
>
> From time to time (2-3 times in a month) I see GC activity in logs and for this time
cassandra stops responding to requests which results in a timeout in upper-layer application.
Total time of unavailability can be over 5 minues (like in the following case)
>
> What can I do with that? Wiil it become much worse when my cluster grows up?
>
> INFO [GossipTasks:1] 2012-07-16 13:10:50,055 Gossiper.java (line 736) InetAddress /10.220.50.9
is now dead.
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,056 GCInspector.java (line 123) GC for ParNew:
391383 ms for 1 collections, 2025808488 used; max is 8464105472
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,067 StatusLogger.java (line 50) Pool Name
Active Pending Blocked
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,071 StatusLogger.java (line 65) ReadStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,071 StatusLogger.java (line 65) RequestResponseStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,072 StatusLogger.java (line 65) ReadRepairStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,072 StatusLogger.java (line 65) MutationStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,073 StatusLogger.java (line 65) ReplicateOnWriteStage
0 0 0
> INFO [GossipStage:1] 2012-07-16 13:10:50,074 Gossiper.java (line 722) InetAddress /10.220.50.9
is now UP
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,075 StatusLogger.java (line 65) GossipStage
1 59 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,075 StatusLogger.java (line 65) AntiEntropyStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,076 StatusLogger.java (line 65) MigrationStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,076 StatusLogger.java (line 65) StreamStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,077 StatusLogger.java (line 65) MemtablePostFlusher
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,077 StatusLogger.java (line 65) FlushWriter
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,077 StatusLogger.java (line 65) MiscStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,078 StatusLogger.java (line 65) InternalResponseStage
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,078 StatusLogger.java (line 65) HintedHandoff
0 0 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,079 StatusLogger.java (line 69) CompactionManager
n/a 0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,079 StatusLogger.java (line 81) MessagingService
n/a 0,0
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,080 StatusLogger.java (line 85) ColumnFamily
Memtable ops,data Row cache size/cap Key cache size/cap
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,080 StatusLogger.java (line 88) Keyspace1.PSS
36712,343842617 0/0 97995/1000000
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,085 StatusLogger.java (line 88) Keyspace1.Standard1
128679,162567721 0/0 0/1000000
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,085 StatusLogger.java (line 88) system.NodeIdInfo
0,0 0/0 0/1
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,086 StatusLogger.java (line 88) system.IndexInfo
0,0 0/0 0/1
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,086 StatusLogger.java (line 88) system.LocationInfo
0,0 0/0 1/2
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,086 StatusLogger.java (line 88) system.Migrations
0,0 0/0 3/3
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,087 StatusLogger.java (line 88) system.HintsColumnFamily
1,528 0/0 0/1
> INFO [ScheduledTasks:1] 2012-07-16 13:10:50,087 StatusLogger.java (line 88) system.Schema
0,0 0/0 2/3
> -------- Завершение пересылаемого сообщения --------
|