cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Roth <benjamin.r...@jaumo.com>
Subject Re: large number of pending compactions, sstables steadily increasing
Date Mon, 07 Nov 2016 20:49:46 GMT
Hm, this MAY somehow relate to the issue I encountered recently:
https://issues.apache.org/jira/browse/CASSANDRA-12730
I also made a proposal to mitigate excessive (unnecessary) flushes during
repair streams but unfortunately nobody commented on it yet.
Maybe there are some opinions on it around here?

2016-11-07 20:15 GMT+00:00 Ben Slater <ben.slater@instaclustr.com>:

> What I’ve seen happen a number of times is you get in a negative feedback
> loop:
> not enough capacity to keep up with compactions (often triggered by repair
> or compaction hitting a large partition) -> more sstables -> more expensive
> reads -> even less capacity to keep up with compactions -> repeat
>
> The way we deal with this at Instaclustr is typically to take the node
> offline to let it catch up with compactions. We take it offline by running
> nodetool disablegossip + disablethrift + disablebinary, unthrottle
> compactions (nodetool setcompactionthroughput 0) and then leave it to chug
> through compactions until it gets close to zero then reverse the settings
> or restart C* to set things back to normal. This typically resolves the
> issues. If you see it happening regularly your cluster probably needs more
> processing capacity (or other tuning).
>
> Cheers
> Ben
>
> On Tue, 8 Nov 2016 at 02:38 Eiti Kimura <eiti.kimura@movile.com> wrote:
>
>> Hey guys,
>>
>> Do we have any conclusions about this case? Ezra, did you solve your
>> problem?
>> We are facing a very similar problem here. LeveledCompaction with VNodes
>> and looks like a node went to a weird state and start to consume lot of
>> CPU, the compaction process seems to be stucked and the number of SSTables
>> increased significantly.
>>
>> Do you have any clue about it?
>>
>> Thanks,
>> Eiti
>>
>>
>>
>> J.P. Eiti Kimura
>> Plataformas
>>
>> +55 19 3518  <https://www.movile.com/assinaturaemail/#>5500
>> + <https://www.movile.com/assinaturaemail/#>55 19 98232 2792
>> skype: eitikimura
>> <https://www.linkedin.com/company/movile>
>> <https://pt.pinterest.com/Movile/>  <https://twitter.com/movile_LATAM>
>> <https://www.facebook.com/Movile>
>>
>> 2016-09-11 18:20 GMT-03:00 Jens Rantil <jens.rantil@tink.se>:
>>
>> I just want to chime in and say that we also had issues keeping up with
>> compaction once (with vnodes/ssd disks) and I also want to recommend
>> keeping track of your open file limit which might bite you.
>>
>> Cheers,
>> Jens
>>
>>
>> On Friday, August 19, 2016, Mark Rose <markrose@markrose.ca> wrote:
>>
>> Hi Ezra,
>>
>> Are you making frequent changes to your rows (including TTL'ed
>> values), or mostly inserting new ones? If you're only inserting new
>> data, it's probable using size-tiered compaction would work better for
>> you. If you are TTL'ing whole rows, consider date-tiered.
>>
>> If leveled compaction is still the best strategy, one way to catch up
>> with compactions is to have less data per partition -- in other words,
>> use more machines. Leveled compaction is CPU expensive. You are CPU
>> bottlenecked currently, or from the other perspective, you have too
>> much data per node for leveled compaction.
>>
>> At this point, compaction is so far behind that you'll likely be
>> getting high latency if you're reading old rows (since dozens to
>> hundreds of uncompacted sstables will likely need to be checked for
>> matching rows). You may be better off with size tiered compaction,
>> even if it will mean always reading several sstables per read (higher
>> latency than when leveled can keep up).
>>
>> How much data do you have per node? Do you update/insert to/delete
>> rows? Do you TTL?
>>
>> Cheers,
>> Mark
>>
>> On Wed, Aug 17, 2016 at 2:39 PM, Ezra Stuetzel <ezra.stuetzel@riskiq.net>
>> wrote:
>> > I have one node in my cluster 2.2.7 (just upgraded from 2.2.6 hoping to
>> fix
>> > issue) which seems to be stuck in a weird state -- with a large number
>> of
>> > pending compactions and sstables. The node is compacting about
>> 500gb/day,
>> > number of pending compactions is going up at about 50/day. It is at
>> about
>> > 2300 pending compactions now. I have tried increasing number of
>> compaction
>> > threads and the compaction throughput, which doesn't seem to help
>> eliminate
>> > the many pending compactions.
>> >
>> > I have tried running 'nodetool cleanup' and 'nodetool compact'. The
>> latter
>> > has fixed the issue in the past, but most recently I was getting OOM
>> errors,
>> > probably due to the large number of sstables. I upgraded to 2.2.7 and
>> am no
>> > longer getting OOM errors, but also it does not resolve the issue. I do
>> see
>> > this message in the logs:
>> >
>> >> INFO  [RMI TCP Connection(611)-10.9.2.218] 2016-08-17 01:50:01,985
>> >> CompactionManager.java:610 - Cannot perform a full major compaction as
>> >> repaired and unrepaired sstables cannot be compacted together. These
>> two set
>> >> of sstables will be compacted separately.
>> >
>> > Below are the 'nodetool tablestats' comparing a normal and the
>> problematic
>> > node. You can see problematic node has many many more sstables, and
>> they are
>> > all in level 1. What is the best way to fix this? Can I just delete
>> those
>> > sstables somehow then run a repair?
>> >>
>> >> Normal node
>> >>>
>> >>> keyspace: mykeyspace
>> >>>
>> >>>     Read Count: 0
>> >>>
>> >>>     Read Latency: NaN ms.
>> >>>
>> >>>     Write Count: 31905656
>> >>>
>> >>>     Write Latency: 0.051713177939359714 ms.
>> >>>
>> >>>     Pending Flushes: 0
>> >>>
>> >>>         Table: mytable
>> >>>
>> >>>         SSTable count: 1908
>> >>>
>> >>>         SSTables in each level: [11/4, 20/10, 213/100, 1356/1000,
>> 306, 0,
>> >>> 0, 0, 0]
>> >>>
>> >>>         Space used (live): 301894591442
>> >>>
>> >>>         Space used (total): 301894591442
>> >>>
>> >>>
>> >>>
>> >>> Problematic node
>> >>>
>> >>> Keyspace: mykeyspace
>> >>>
>> >>>     Read Count: 0
>> >>>
>> >>>     Read Latency: NaN ms.
>> >>>
>> >>>     Write Count: 30520190
>> >>>
>> >>>     Write Latency: 0.05171286705620116 ms.
>> >>>
>> >>>     Pending Flushes: 0
>> >>>
>> >>>         Table: mytable
>> >>>
>> >>>         SSTable count: 14105
>> >>>
>> >>>         SSTables in each level: [13039/4, 21/10, 206/100, 831, 0, 0,
>> 0,
>> >>> 0, 0]
>> >>>
>> >>>         Space used (live): 561143255289
>> >>>
>> >>>         Space used (total): 561143255289
>> >
>> > Thanks,
>> >
>> > Ezra
>>
>>
>>
>> --
>> Jens Rantil
>> Backend engineer
>> Tink AB
>>
>> Email: jens.rantil@tink.se
>> Phone: +46 708 84 18 32
>> Web: www.tink.se
>>
>> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
>> <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
>>  Twitter <https://twitter.com/tink>
>>
>>
>>


-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Mime
View raw message