cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Griffith (JIRA)" <>
Subject [jira] [Created] (CASSANDRA-10515) Compaction hangs with move to 2.1.10
Date Tue, 13 Oct 2015 19:26:06 GMT
Jeff Griffith created CASSANDRA-10515:

             Summary: Compaction hangs with move to 2.1.10
                 Key: CASSANDRA-10515
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: redhat 6.5, cassandra 2.1.10
            Reporter: Jeff Griffith
            Priority: Critical

After upgrading from cassandra 2.0.x to 2.1.10, we began seeing problems where some nodes
break the 12G commit log max we configured and go as high as 65G or more. Once it reaches
this state, "nodetool compactionstats" hangs. I watched the recovery live when compactions
begin happening again. the "nodetool compactionstats" suddenly completed to show the outstanding
jobs most in 100% completion state:

jgriffith@prod1xc1.c2.bf1:~$ ndc
pending tasks: 2185
   compaction type   keyspace                          table     completed          total
   unit   progress
        Compaction   SyncCore      ContactInformationUpdates   61251208033   170643574558
  bytes     35.89%
        Compaction   SyncCore                     CommEvents   19262483904    19266079916
  bytes     99.98%
        Compaction   SyncCore   EndpointPrefixIndexMinimized    6592197093     6592316682
  bytes    100.00%
        Compaction   SyncCore           EmailHistogramDeltas    3411039555     3411039557
  bytes    100.00%
        Compaction   SyncCore        ContactPrefixBytesIndex    2879241009     2879487621
  bytes     99.99%
        Compaction   SyncCore               EndpointProfiles   21252493623    21252635196
  bytes    100.00%
        Compaction   SyncCore                     CommEvents   81009853587    81009854438
  bytes    100.00%
        Compaction   SyncCore             EndpointIndexIntId    3005734580     3005768582
  bytes    100.00%
Active compaction remaining time :        n/a

I was also doing periodic "nodetool tpstats" which were working but not being logged in system.log
on the StatusLogger thread until after the compaction started working again.

This message was sent by Atlassian JIRA

View raw message