cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra 1.0 hangs during GC
Date Wed, 18 Jul 2012 03:51:45 GMT
Assuming all the memory and yaml settings default that does not sound right. 

The first thought would be the memory meter not counting correctly...

Do you do a lot of deletes ? 

Do you have a lot of CF's and/or secondary indexes ? 

Can you see log lines about the "liveRatio" for your cf's ? 

I would upgrade to 1.0.10 before getting too carried away though. 

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 17/07/2012, at 8:14 PM, Nikolay Kоvshov wrote:

> 
> This is a cluster of 2 nodes, each having 8G of operating memory, replicationfactor=2
> Write/read pressure is quite low and almost never exceeds 10/second
> 
> From time to time (2-3 times in a month) I see GC activity in logs and for this time
cassandra stops responding to requests which results in a timeout in upper-layer application.
Total time of unavailability can be over 5 minues (like in the following case)
> 
> What can I do with that? Wiil it become much worse when my cluster grows up?
> 
> INFO [GossipTasks:1] 2012-07-16 13:10:50,055 Gossiper.java (line 736) InetAddress /10.220.50.9
is now dead.
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,056 GCInspector.java (line 123) GC for ParNew:
391383 ms for 1 collections, 2025808488 used; max is 8464105472
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,067 StatusLogger.java (line 50) Pool Name
                   Active   Pending   Blocked
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,071 StatusLogger.java (line 65) ReadStage
                        0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,071 StatusLogger.java (line 65) RequestResponseStage
             0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,072 StatusLogger.java (line 65) ReadRepairStage
                  0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,072 StatusLogger.java (line 65) MutationStage
                    0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,073 StatusLogger.java (line 65) ReplicateOnWriteStage
            0         0         0
>  INFO [GossipStage:1] 2012-07-16 13:10:50,074 Gossiper.java (line 722) InetAddress /10.220.50.9
is now UP
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,075 StatusLogger.java (line 65) GossipStage
                      1        59         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,075 StatusLogger.java (line 65) AntiEntropyStage
                 0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,076 StatusLogger.java (line 65) MigrationStage
                   0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,076 StatusLogger.java (line 65) StreamStage
                      0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,077 StatusLogger.java (line 65) MemtablePostFlusher
              0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,077 StatusLogger.java (line 65) FlushWriter
                      0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,077 StatusLogger.java (line 65) MiscStage
                        0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,078 StatusLogger.java (line 65) InternalResponseStage
            0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,078 StatusLogger.java (line 65) HintedHandoff
                    0         0         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,079 StatusLogger.java (line 69) CompactionManager
              n/a         0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,079 StatusLogger.java (line 81) MessagingService
               n/a       0,0
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,080 StatusLogger.java (line 85) ColumnFamily
               Memtable ops,data  Row cache size/cap  Key cache size/cap
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,080 StatusLogger.java (line 88) Keyspace1.PSS
                36712,343842617                 0/0       97995/1000000
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,085 StatusLogger.java (line 88) Keyspace1.Standard1
         128679,162567721                 0/0           0/1000000
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,085 StatusLogger.java (line 88) system.NodeIdInfo
                        0,0                 0/0                 0/1
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,086 StatusLogger.java (line 88) system.IndexInfo
                         0,0                 0/0                 0/1
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,086 StatusLogger.java (line 88) system.LocationInfo
                      0,0                 0/0                 1/2
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,086 StatusLogger.java (line 88) system.Migrations
                        0,0                 0/0                 3/3
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,087 StatusLogger.java (line 88) system.HintsColumnFamily
               1,528                 0/0                 0/1
>  INFO [ScheduledTasks:1] 2012-07-16 13:10:50,087 StatusLogger.java (line 88) system.Schema
                            0,0                 0/0                 2/3
> -------- Завершение пересылаемого сообщения --------


Mime
View raw message