incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Theroux <mthero...@yahoo.com>
Subject Cassandra Messages Dropped
Date Mon, 17 Sep 2012 14:16:32 GMT
Hello,

While under load, we have occasionally been seeing "messages dropped" errors in our cassandra
log.  Doing some research, I understand this is part of Cassandra's design to shed load, and
we should look at the tpstats-like output to determine what should be done to resolve the
situation.  Typically, you will see lots of messages blocked or pending, and that might be
an indicator that a specific part of hardware needs to be improved/tuned/upgraded.  

However, looking at the output we are getting, I'm finding it difficult to see what needs
to be tuned, as it looks to me cassandra is handling the load within the mutation stage:

NFO [ScheduledTasks:1] 2012-09-17 06:28:03,266 MessagingService.java (line 658) 119 MUTATION
messages dropped in last 5000ms
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,645 StatusLogger.java (line 57) Pool Name   
                Active   Pending   Blocked
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,836 StatusLogger.java (line 72) ReadStage   
                     3         3         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,837 StatusLogger.java (line 72) RequestResponseStage
             0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,837 StatusLogger.java (line 72) ReadRepairStage
                  0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,837 StatusLogger.java (line 72) MutationStage
                    0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,838 StatusLogger.java (line 72) ReplicateOnWriteStage
            0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,838 StatusLogger.java (line 72) GossipStage 
                     0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) AntiEntropyStage
                 0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) MigrationStage
                   0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) StreamStage 
                     0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) MemtablePostFlusher
              1         5         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,840 StatusLogger.java (line 72) FlushWriter 
                     1         5         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,840 StatusLogger.java (line 72) MiscStage   
                     0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,840 StatusLogger.java (line 72) commitlog_archiver
               0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,841 StatusLogger.java (line 72) InternalResponseStage
            0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,841 StatusLogger.java (line 72) AntiEntropySessions
              0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,851 StatusLogger.java (line 72) HintedHandoff
                    0         0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,851 StatusLogger.java (line 77) CompactionManager
                0         0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,852 StatusLogger.java (line 89) MessagingService
               n/a       0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,852 StatusLogger.java (line 99) Cache Type  
                  Size                 Capacity               KeysToSave                 
                                       Provider
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,853 StatusLogger.java (line 100) KeyCache   
                2184533                  2184533                      all                
                                                
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,853 StatusLogger.java (line 106) RowCache   
                      0                        0                      all              org.apache.cassandra.cache.SerializingCacheProvider
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,853 StatusLogger.java (line 113) ColumnFamily
               Memtable ops,data
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,853 StatusLogger.java (line 116) system.NodeIdInfo
                        0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,854 StatusLogger.java (line 116) system.IndexInfo
                         0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,854 StatusLogger.java (line 116) system.LocationInfo
                      0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,854 StatusLogger.java (line 116) system.Versions
                          0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,855 StatusLogger.java (line 116) system.schema_keyspaces
                  0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,855 StatusLogger.java (line 116) system.Migrations
                        0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,855 StatusLogger.java (line 116) system.schema_columnfamilies
                0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,856 StatusLogger.java (line 116) system.schema_columns
                    0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,856 StatusLogger.java (line 116) system.HintsColumnFamily
                 0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,856 StatusLogger.java (line 116) system.Schema
                            0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,857 StatusLogger.java (line 116) prod.comp  
                        0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,857 StatusLogger.java (line 116) prod.byp   
                  224,61863
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,857 StatusLogger.java (line 116) prod.byn   
                  30,6106
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,857 StatusLogger.java (line 116) prod.p     
               3354,3792954
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,858 StatusLogger.java (line 116) prod.rel   
            788,153294
INFO [ScheduledTasks:1] 2012-09-17 06:28:03,858 StatusLogger.java (line 116) prod.i      
                 46,271517
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,859 StatusLogger.java (line 116) prod.users 
                   18489,5650405
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,859 StatusLogger.java (line 116) prod.caches
                      40,7512998
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,859 StatusLogger.java (line 116) prod.sessions
                 8085,13063121
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,859 StatusLogger.java (line 116) prod.content
                            0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,860 StatusLogger.java (line 116) prod.enr   
              436,341190
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,860 StatusLogger.java (line 116) prod.byt   
             23800,9880339
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,861 StatusLogger.java (line 116) prod.aliases
                 63603,10475769
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,861 StatusLogger.java (line 116) prod.ran   
                1500,329387
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,861 StatusLogger.java (line 116) prod.cmap  
                     0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,861 StatusLogger.java (line 116) prod.pmap  
             73032,61237450
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,862 StatusLogger.java (line 116) prod.pic   
               5421,4438066
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,862 StatusLogger.java (line 116) prod.watchdog
                           0,0
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,863 StatusLogger.java (line 116) prod.s     
                   31,40924
 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,863 StatusLogger.java (line 116) prod.m     
           21690,64262284


We are using a write consistency level of LOCAL_QUROUM over a six node cluster.  Any observations
are welcome,

Thanks,
-Mike
Mime
View raw message