cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arne Claassen <a...@emotient.com>
Subject Re: 100% CPU utilization, ParNew and never completing compactions
Date Tue, 16 Dec 2014 19:36:55 GMT
I'm running 2.0.10.

The data is all time series data and as we change our pipeline, we've been
periodically been reprocessing the data sources, which causes each time
series to be overwritten, i.e. every row per partition key is deleted and
re-written, so I assume i've been collecting a bunch of tombstones.

Also, the presence of the ever present and never completing compaction
types, i assumed were an artifact of tombstoning, but i fully admit to
conjecture based on about ~20 blog posts and stackoverflow questions i've
surveyed.

I doubled the Heap on one node and it changed nothing regarding the load or
the ParNew log statements. New Generation Usage is 50%, Eden itself is 56%.

Anything else i should look at and report, let me know.

On Tue, Dec 16, 2014 at 11:14 AM, Jonathan Lacefield <
jlacefield@datastax.com> wrote:
>
> Hello,
>
>   What version of Cassandra are you running?
>
>   If it's 2.0, we recently experienced something similar with 8447 [1],
> which 8485 [2] should hopefully resolve.
>
>   Please note that 8447 is not related to tombstones.  Tombstone
> processing can put a lot of pressure on the heap as well. Why do you think
> you have a lot of tombstones in that one particular table?
>
>   [1] https://issues.apache.org/jira/browse/CASSANDRA-8447
>   [2] https://issues.apache.org/jira/browse/CASSANDRA-8485
>
> Jonathan
>
> [image: datastax_logo.png]
>
> Jonathan Lacefield
>
> Solution Architect | (404) 822 3487 | jlacefield@datastax.com
>
> [image: linkedin.png] <http://www.linkedin.com/in/jlacefield/> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax> <https://github.com/datastax/>
>
> On Tue, Dec 16, 2014 at 2:04 PM, Arne Claassen <arne@emotient.com> wrote:
>>
>> I have a three node cluster that has been sitting at a load of 4 (for
>> each node), 100% CPI utilization (although 92% nice) for that last 12
>> hours, ever since some significant writes finished. I'm trying to determine
>> what tuning I should be doing to get it out of this state. The debug log is
>> just an endless series of:
>>
>> DEBUG [ScheduledTasks:1] 2014-12-16 19:03:35,042 GCInspector.java (line
>> 118) GC for ParNew: 166 ms for 10 collections, 4400928736 used; max is
>> 8000634880
>> DEBUG [ScheduledTasks:1] 2014-12-16 19:03:36,043 GCInspector.java (line
>> 118) GC for ParNew: 165 ms for 10 collections, 4440011176 used; max is
>> 8000634880
>> DEBUG [ScheduledTasks:1] 2014-12-16 19:03:37,043 GCInspector.java (line
>> 118) GC for ParNew: 135 ms for 8 collections, 4402220568 used; max is
>> 8000634880
>>
>> iostat shows virtually no I/O.
>>
>> Compaction may enter into this, but i don't really know what to make of
>> compaction stats since they never change:
>>
>> [root@cassandra-37919c3a ~]# nodetool compactionstats
>> pending tasks: 10
>>           compaction type        keyspace           table       completed
>>           total      unit  progress
>>                Compaction           mediamedia_tracks_raw       271651482
>>       563615497     bytes    48.20%
>>                Compaction           mediamedia_tracks_raw        30308910
>>     21676695677     bytes     0.14%
>>                Compaction           mediamedia_tracks_raw      1198384080
>>      1815603161     bytes    66.00%
>> Active compaction remaining time :   0h22m24s
>>
>> 5 minutes later:
>>
>> [root@cassandra-37919c3a ~]# nodetool compactionstats
>> pending tasks: 9
>>           compaction type        keyspace           table       completed
>>           total      unit  progress
>>                Compaction           mediamedia_tracks_raw       271651482
>>       563615497     bytes    48.20%
>>                Compaction           mediamedia_tracks_raw        30308910
>>     21676695677     bytes     0.14%
>>                Compaction           mediamedia_tracks_raw      1198384080
>>      1815603161     bytes    66.00%
>> Active compaction remaining time :   0h22m24s
>>
>> Sure the pending tasks went down by one, but the rest is identical.
>> media_tracks_raw likely has a bunch of tombstones (can't figure out how to
>> get stats on that).
>>
>> Is this behavior something that indicates that i need more Heap, larger
>> new generation? Should I be manually running compaction on tables with lots
>> of tombstones?
>>
>> Any suggestions or places to educate myself better on performance tuning
>> would be appreciated.
>>
>> arne
>>
>

Mime
View raw message