cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuo Chen <chenatu2...@gmail.com>
Subject Re: Too many compactions, maybe keyspace system?
Date Sat, 09 Jan 2016 16:23:44 GMT
I don't know what exactly compaction logs is like in system.log. But I see
logs like this in system.log, I think maybe this is the compaction log

INFO [ScheduledTasks:1] 2016-01-09 22:51:02,664 StatusLogger.java (line
118) ColumnFamily                Memtable ops,data
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.schema_triggers                    0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.local                              0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.peers                              0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.batchlog                           0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.NodeIdInfo                         0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.compactions_in_progress                 0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.schema_keyspaces                   0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.compaction_history              9,2650
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.paxos                              0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.schema_columns                     0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.schema_columnfamilies                 0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.IndexInfo                          0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.range_xfers                        0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.peer_events                        0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.hints                              0,0
 INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.sstable_activity              69,20930
...

To clarify the source of compaction, I restarted the cluster and truncate
system.hints at Jan 8. However, it also shows compactionhistory of hints
like this. I checked the timestamp however, many history items are compared
at Jan 6.

-b313-11e5-a211-45b7aa88107c     system             hints
     1451933296208             14472833       14276596       {1:2}
7a7e5ad0-b42c-11e5-a211-45b7aa88107c     system
sstable_activity             1452053883133             3215           852
         {4:23}
16e97320-b264-11e5-a211-45b7aa88107c     system
sstable_activity             1451857865554             3114           726
         {4:23}
f8ab7d40-b62d-11e5-aa30-45b7aa88107c     system
sstable_activity             1452274426644             3047           751
         {4:23}
ea955a00-b173-11e5-a211-45b7aa88107c     system             hints
             1451754711968             18970740       18970740       {1:1}
cf98c3c0-b54e-11e5-bce5-45b7aa88107c     system
sstable_activity             1452178579964             3503           964
         {4:23}
9e74bf80-b4c0-11e5-ac1a-45b7aa88107c     system
sstable_activity             1452117508984             3135           800
         {4:22}
4b9ffb20-b27d-11e5-a211-45b7aa88107c     system
sstable_activity             1451868691410             3037           791
         {4:23}
4f059400-b3e0-11e5-a211-45b7aa88107c     system
sstable_activity             1452021168448             2985           802
         {1:2, 4:23}
9de3b670-b439-11e5-ac1a-45b7aa88107c     system             hints
             1452059525975             10532201       10550449       {2:2}
f1c2f710-b439-11e5-ac1a-45b7aa88107c     system             hints
             1452059666689             10479415       10479415       {1:1}
9b4f7700-b439-11e5-ac1a-45b7aa88107c     system             hints
             1452059521648             10594437       10532000       {1:1,
2:1}
402f42d0-b2fc-11e5-a211-45b7aa88107c     system
sstable_activity             1451923218301             3184           776
         {4:23}
47589f00-b231-11e5-a211-45b7aa88107c     system
sstable_activity             1451836042480             3212           785
         {4:23}
848ee140-b439-11e5-ac1a-45b7aa88107c     system             hints
             1452059483476             10648367       10601270       {1:1,
2:1}
8736f390-b4f9-11e5-ac1a-45b7aa88107c     system             hints
             1452141951305             8261575        8112577        {1:1}
ff591b80-b37a-11e5-a211-45b7aa88107c     system
sstable_activity             1451977655608             3025           765
         {4:23}

Besides, there are also lots pending compaction tasks over 70000000.

On Sat, Jan 9, 2016 at 2:42 AM, Robert Coli <rcoli@eventbrite.com> wrote:

>
>
> On Thu, Jan 7, 2016 at 6:30 PM, Shuo Chen <chenatu2006@gmail.com> wrote:,
>>
>> I am using Cassandra 2.0.16 with 4 nodes and found too many compactions
>> for this cluster. This caused too much full gc and choked the system. I
>> have discussed the high gc in previous mails but didnot get the satisfied
>> answers.
>>
>> To clarify the source of the compactions, I shutdown all the clients and
>> there is no reading and writing requests outside. Besides,
>>  hinted_handoff_enabled: false is set. The cluster is restarted yesterday
>> and have run for 1.5 days.
>>
>
> Do you see compactions of the hints CF in the system.log?
>
> =Rob
>
>



-- 
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com

Mime
View raw message