cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paddy Krishnamoorthy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-9331) Too many Compaction pending Tasks
Date Fri, 08 May 2015 03:52:59 GMT
Paddy Krishnamoorthy created CASSANDRA-9331:
-----------------------------------------------

             Summary: Too many Compaction pending Tasks
                 Key: CASSANDRA-9331
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9331
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: Ubuntu
            Reporter: Paddy Krishnamoorthy
            Priority: Critical
             Fix For: 2.1.x


We have a 9 node cluster that has been running for about 8 months now. Lately the selects
seems to take longer. Also I noticed that the compaction jobs are always running. Last time
when I checked every node has average 600+ pending compaction tasks. Two days ago I posted
this issue on IRC chat and someone recommended to upgrade from 2.1.2 to current release. I
updated one of the nodes to 2.1.4 and I still noticed that newly added node has about 600+
compaction tasks.  We also noticed that "nodetool rebuild"  takes more than 24 hours to complete.
Wondering if there is an issue with our configuration or is it a bug?

Hardware and other config details:
	The current storage size per node is approximately 100GB. 
	All our disks are spinning disks. 
	The servers are hosted in out own data centre. 
	We use VNODES and num_tokens set to 256

Here is the out from "desc table daily_index" command for one of our table that has 10s of
millions of records. 

CREATE TABLE daily_indexes.daily_index ( indexday int, pid text, docid text, author text,
e map<int, decimal>, hit boolean,     p decimal,     s int,     PRIMARY KEY ((indexday,
pid), docid) ) WITH CLUSTERING ORDER BY (docid ASC)     AND bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'     AND comment = ''     AND
compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
'max_threshold': '32'}     AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1  AND default_time_to_live = 0     AND gc_grace_seconds
= 86400     AND max_index_interval = 2048     AND memtable_flush_period_in_ms = 0     AND
min_index_interval = 128     AND read_repair_chance = 0.0     AND speculative_retry = '99.0PERCENTILE';






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message