cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Simonov <dimmobor...@gmail.com>
Subject Re: Many SSTables only on one node
Date Thu, 05 Apr 2018 09:33:13 GMT
Hi, Alexander!

SizeTieredCompactionStrategy is used for all CFs in problematic keyspace.
Current compaction throughput is 16 MB/s (default value).

We always have about 40 pending and 2 active "CompactionExecutor" tasks in
"tpstats".
Mostly because of another (bigger) keyspace in this cluster.
But the situation is the same on each node.

According to "nodetool compactionhistory", compactions on this CF run
(sometimes several times per day, sometimes one time per day, the last run
was yesterday).
We run "repair -full" regulary for this keyspace (every 24 hours on each
node), because gc_grace_seconds is set to 24 hours.

Should we consider increasing compaction throughput and
"concurrent_compactors" (as recommended for SSDs) to keep
"CompactionExecutor" pending tasks low?

2018-04-05 14:09 GMT+05:00 Alexander Dejanovski <alex@thelastpickle.com>:

> Hi Dmitry,
>
> could you tell us which compaction strategy that table is currently using ?
> Also, what is the compaction max throughput and is auto-compaction
> correctly enabled on that node ?
>
> Did you recently run repair ?
>
> Thanks,
>
> On Thu, Apr 5, 2018 at 10:53 AM Dmitry Simonov <dimmoborgir@gmail.com>
> wrote:
>
>> Hello!
>>
>> Could you please give some ideas on the following problem?
>>
>> We have a cluster with 3 nodes, running Cassandra 2.2.11.
>>
>> We've recently discovered high CPU usage on one cluster node, after some
>> investigation we found that number of sstables for one CF on it is very
>> big: 5800 sstables, on other nodes: 3 sstable.
>>
>> Data size in this keyspace was not very big ~100-200Mb per node.
>>
>> There is no such problem with other CFs of that keyspace.
>>
>> nodetool compact solved the issue as a quick-fix.
>>
>> But I'm wondering, what was the cause? How prevent it from repeating?
>>
>> --
>> Best Regards,
>> Dmitry Simonov
>>
> --
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>



-- 
Best Regards,
Dmitry Simonov

Mime
View raw message