cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "graham sanderson (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
Date Mon, 21 Jul 2014 19:04:42 GMT


graham sanderson updated CASSANDRA-7546:

    Attachment: 7546.20_2.txt

Yes, I agree with you; simpler is better - I didn't like the fact that it was hard to disentangle
my code.

So I have attached a different patch 7546.20_2.txt, which is about the same as yours except
that it tracks a very minimal state heuristic to avoid the first unlocked loop when it thinks
it is contended... this helps under load where it attempts about 1.11 loop per mutation in
the heavily contented case vs 1.93 with 7546.20.txt (or about 18 without any patch at all).

It turns out to be rather hard to measure this with cassandra-stress in isolation - basically
the looping causes very high memory allocation rates which may or may not be a problem based
on the current state of memory in the JVM (e.g. currently resident recent memtables) and other
activity. Also, I may open a separate ticket to allow batch (thrift and maybe CQL) inserts
in cassandra-stress to bump up the load a little more.

That said, clearly this race is undesirable, and we have certainly observed its effects in
practice in the hinting case (which is doubly unfortunate since you are already now doing
more work per node than you thought, and you are going down a contention path you weren't
expecting based on your own partitioning). Note when it goes wrong it goes really wrong!

So, I think settling for a patch with minimal code or uncontended path impact, that caps us
nicely is the right way to go

> AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
> -----------------------------------------------------------------------------
>                 Key: CASSANDRA-7546
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: graham sanderson
>            Assignee: graham sanderson
>         Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_alt.txt, suggestion1.txt, suggestion1_21.txt
> In order to preserve atomicity, this code attempts to read, clone/update, then CAS the
state of the partition.
> Under heavy contention for updating a single partition this can cause some fairly staggering
memory growth (the more cores on your machine the worst it gets).
> Whilst many usage patterns don't do highly concurrent updates to the same partition,
hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory
allocation rates can be seen (especially when the updates being hinted are small updates to
different partitions which can happen very fast on their own) - see CASSANDRA-7545
> It would be best to eliminate/reduce/limit the spinning memory allocation whilst not
slowing down the very common un-contended case.

This message was sent by Atlassian JIRA

View raw message