cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eiti Kimura <eiti.kim...@movile.com>
Subject Re: large number of pending compactions, sstables steadily increasing
Date Mon, 07 Nov 2016 15:37:54 GMT
Hey guys,

Do we have any conclusions about this case? Ezra, did you solve your
problem?
We are facing a very similar problem here. LeveledCompaction with VNodes
and looks like a node went to a weird state and start to consume lot of
CPU, the compaction process seems to be stucked and the number of SSTables
increased significantly.

Do you have any clue about it?

Thanks,
Eiti



J.P. Eiti Kimura
Plataformas

+55 19 3518  <https://www.movile.com/assinaturaemail/#>5500
+ <https://www.movile.com/assinaturaemail/#>55 19 98232 2792
skype: eitikimura
<https://www.linkedin.com/company/movile>
<https://pt.pinterest.com/Movile/>  <https://twitter.com/movile_LATAM>
<https://www.facebook.com/Movile>

2016-09-11 18:20 GMT-03:00 Jens Rantil <jens.rantil@tink.se>:

> I just want to chime in and say that we also had issues keeping up with
> compaction once (with vnodes/ssd disks) and I also want to recommend
> keeping track of your open file limit which might bite you.
>
> Cheers,
> Jens
>
>
> On Friday, August 19, 2016, Mark Rose <markrose@markrose.ca> wrote:
>
>> Hi Ezra,
>>
>> Are you making frequent changes to your rows (including TTL'ed
>> values), or mostly inserting new ones? If you're only inserting new
>> data, it's probable using size-tiered compaction would work better for
>> you. If you are TTL'ing whole rows, consider date-tiered.
>>
>> If leveled compaction is still the best strategy, one way to catch up
>> with compactions is to have less data per partition -- in other words,
>> use more machines. Leveled compaction is CPU expensive. You are CPU
>> bottlenecked currently, or from the other perspective, you have too
>> much data per node for leveled compaction.
>>
>> At this point, compaction is so far behind that you'll likely be
>> getting high latency if you're reading old rows (since dozens to
>> hundreds of uncompacted sstables will likely need to be checked for
>> matching rows). You may be better off with size tiered compaction,
>> even if it will mean always reading several sstables per read (higher
>> latency than when leveled can keep up).
>>
>> How much data do you have per node? Do you update/insert to/delete
>> rows? Do you TTL?
>>
>> Cheers,
>> Mark
>>
>> On Wed, Aug 17, 2016 at 2:39 PM, Ezra Stuetzel <ezra.stuetzel@riskiq.net>
>> wrote:
>> > I have one node in my cluster 2.2.7 (just upgraded from 2.2.6 hoping to
>> fix
>> > issue) which seems to be stuck in a weird state -- with a large number
>> of
>> > pending compactions and sstables. The node is compacting about
>> 500gb/day,
>> > number of pending compactions is going up at about 50/day. It is at
>> about
>> > 2300 pending compactions now. I have tried increasing number of
>> compaction
>> > threads and the compaction throughput, which doesn't seem to help
>> eliminate
>> > the many pending compactions.
>> >
>> > I have tried running 'nodetool cleanup' and 'nodetool compact'. The
>> latter
>> > has fixed the issue in the past, but most recently I was getting OOM
>> errors,
>> > probably due to the large number of sstables. I upgraded to 2.2.7 and
>> am no
>> > longer getting OOM errors, but also it does not resolve the issue. I do
>> see
>> > this message in the logs:
>> >
>> >> INFO  [RMI TCP Connection(611)-10.9.2.218] 2016-08-17 01:50:01,985
>> >> CompactionManager.java:610 - Cannot perform a full major compaction as
>> >> repaired and unrepaired sstables cannot be compacted together. These
>> two set
>> >> of sstables will be compacted separately.
>> >
>> > Below are the 'nodetool tablestats' comparing a normal and the
>> problematic
>> > node. You can see problematic node has many many more sstables, and
>> they are
>> > all in level 1. What is the best way to fix this? Can I just delete
>> those
>> > sstables somehow then run a repair?
>> >>
>> >> Normal node
>> >>>
>> >>> keyspace: mykeyspace
>> >>>
>> >>>     Read Count: 0
>> >>>
>> >>>     Read Latency: NaN ms.
>> >>>
>> >>>     Write Count: 31905656
>> >>>
>> >>>     Write Latency: 0.051713177939359714 ms.
>> >>>
>> >>>     Pending Flushes: 0
>> >>>
>> >>>         Table: mytable
>> >>>
>> >>>         SSTable count: 1908
>> >>>
>> >>>         SSTables in each level: [11/4, 20/10, 213/100, 1356/1000,
>> 306, 0,
>> >>> 0, 0, 0]
>> >>>
>> >>>         Space used (live): 301894591442
>> >>>
>> >>>         Space used (total): 301894591442
>> >>>
>> >>>
>> >>>
>> >>> Problematic node
>> >>>
>> >>> Keyspace: mykeyspace
>> >>>
>> >>>     Read Count: 0
>> >>>
>> >>>     Read Latency: NaN ms.
>> >>>
>> >>>     Write Count: 30520190
>> >>>
>> >>>     Write Latency: 0.05171286705620116 ms.
>> >>>
>> >>>     Pending Flushes: 0
>> >>>
>> >>>         Table: mytable
>> >>>
>> >>>         SSTable count: 14105
>> >>>
>> >>>         SSTables in each level: [13039/4, 21/10, 206/100, 831, 0, 0,
>> 0,
>> >>> 0, 0]
>> >>>
>> >>>         Space used (live): 561143255289
>> >>>
>> >>>         Space used (total): 561143255289
>> >
>> > Thanks,
>> >
>> > Ezra
>>
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
>  Twitter <https://twitter.com/tink>
>
>

Mime
View raw message