cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: High CPU usage on reading single row with Set column with short TTL
Date Mon, 28 Jan 2019 16:24:28 GMT
Probably lowest effort is to run the select with tracing enabled - may give some easy hints


-- 
Jeff Jirsa


> On Jan 28, 2019, at 7:54 AM, Jonathan Haddad <jon@jonhaddad.com> wrote:
> 
> Your fastest route might be to run a profiler on Cassandra and get some flame graphs.
 I'm a fan of the async-profiler:
> 
> https://github.com/jvm-profiling-tools/async-profiler
> 
> Joey Lynch did a nice write up in the documentation on a different process, which I haven't
used yet:
> 
> http://cassandra.apache.org/doc/latest/troubleshooting/use_tools.html#cpu-flamegraphs
> 
> If I have some time today I'll put together a tlp-stress (https://github.com/thelastpickle/tlp-stress)
workload to see if I can reproduce it locally.
> 
> Jon
> 
> 
> 
>> On Mon, Jan 28, 2019 at 7:23 AM Tom Wollert <tomwollert@codeweavers.net> wrote:
>> 
>> Hello,
>> 
>> We have noticed CPU usage spike after several minutes of consistent load  when querying:
>> - a single column of set<uuid> type (same partition key) 
>> - relatively frequently (couple hundred times per second, for comparison, we do an
order of magnitude more reads already with much bigger payloads)
>> - with the elements in the set having a very short TTL ( single digit seconds) and
several inserts per second
>> - gc_grace set to 0 (that should remove hints and should prevent tombstones)
>> - reads and writes are using local quorum consistency
>> - replication factor of 3 (on 4 node setup)
>> 
>> I am struggling to figure out where the high CPU usage comes from (and thus how to
resolve it) and hoping that some one sees what we are doing wrong. I'd expect the data to
stay in memory on the cluster and have constant read time.
>> 
>> The use case is rate limiting. We basically limit a user (for example) to 20 requests
per 5 seconds and are using cassandra's TTL to implement it across all live servers. So when
a request comes along we run the following query:
>> 
>> SELECT tokens
>> FROM recent_request_token_bucket 
>> WHERE  usagekey = 'some user id'
>> 
>> If the tokens' count is less than 20 we execute
>> UPDATE recent_request_token_bucket 
>> USING TTL 5
>> SET tokens = tokens + 'Guid.NewGuid()' 
>> WHERE usagekey =  'some user id'
>> 
>> If the token's count is greater than 20 we reject the request
>> 
>> The table definition is 
>> CREATE TABLE recent_request_token_bucket 
>>             (
>>                 usagekey text,
>>                 tokens set<uuid>,
>>                 PRIMARY KEY (usagekey)
>>             )
>>             WITH
>>                 compaction={'min_threshold': 2, 'class': 'SizeTieredCompactionStrategy',
'max_threshold': 32}
>>             AND
>>                 compression={'sstable_compression': 'SnappyCompressor'}
>>             AND 
>>                 gc_grace_seconds=0;
>> 
>> I have replicated it with the following:
>> 200 reads per second
>> 3 inserts per second
>> 
>> This starts of with CPU load is ~10% and average response time (as reported by my
console app) 1-2 ms
>> After 5 the CPU load creeps up to ~20% and average response time 2-4ms
>> After 10 minutes the CPU load is over 50 and average response times starts to hit
10ms
>> After 15 minutes the CPU load is near 100% and response times over 100ms become normal.
>> 
>> Interestingly, when aborting the application, waiting several minutes and then restarting,
the response times and CPU load on the server remain terrible. It's like I poisoned that partition
key permanently. This also survives flushes of the memtable. 
>> 
>> I'd expect a constant response time in our use case as there should be no more than
20 odd guids in the set. But it appears that cassandra maintains the tombstones in memory?

>> 
>> We are running 2.1.20 
>> 
>> I'd appreciate any pointers!
>> 
>> Cheers,
>> 
>> Tom
>> 
>> -- 
>> Development Director
>> 
>> | T: 0800 021 0888 | M: 0790 4489797 | www.codeweavers.net |
>> | Codeweavers Limited | Barn 4 | Dunston Business Village | Dunston | ST18 9AB |
>> | Registered in England and Wales No. 04092394 | VAT registration no. 974 9705 63
|
>> 
>>  CUSTOMERS' BLOG     TWITTER       FACEBOOK      LINKED IN     DEVELOPERS' BLOG 
    YOUTUBE
>> 
>> 
>> 
>> What's happened at Codeweavers in 2018? l Codeweavers 2018 Stats & Trends
>> 
>> Phone: 0800 021 0888   Email: contactus@codeweavers.net
>> Codeweavers Ltd | Barn 4 | Dunston Business Village | Dunston | ST18 9AB
>> Registered in England and Wales No. 04092394 | VAT registration no. 974 9705 63 
>> 
>>   
> 
> 
> -- 
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade

Mime
View raw message