cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "graham sanderson (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
Date Fri, 25 Jul 2014 06:10:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074121#comment-14074121
] 

graham sanderson edited comment on CASSANDRA-7546 at 7/25/14 6:09 AM:
----------------------------------------------------------------------

Ok, added 7546.20_4.txt patch against 2.0 (with a bit of static debug stuff we probably don't
need)

Currently it is set to 10M + 7.5M/s just because it is cleaner with a power of 2 clock divider...
We could tweak these, but this worked well in my tests both on my laptop and on one of our
production class machines. It basically did exactly as you'd expect which was to kick in once
memory was actually being wasted at a reasonable rate.

A few random thoughts

# It would be nice to expose a metric on the CF (I'm not sure whether it should be just total
counts, or some fraction)... it'd need to be dealt with maybe in {{Memtable.FlushRunnable}}...
I left a package protected method {{usePessimisticLocking}}. I need your advice on whether/how
to expose this, but it'd be nice to watch in test (possibly with the actualy synchronization
turned off) to see how much it is triggered
# Note that we are actually running with {{concurrent_writes = 256}} in production as we have
lots of cores and very fast IO... so it seems likely that we can keep a lot of cores contended
in the hinting case... At a minimum of the rule of thumb would have put it at 128 which would
still be plenty high - I just thought I'd point that out
# I got rid of the unsafe again since we don't need it. I made a synchronized method rather
than plopping some synchronized blocks inside the method bodies because c2 compiler used to
be a bit reticent to inline methods with monitor usage and so it is kind of a habit for now
(also in this case it is probably just as clean if not cleaner).
# Oh, as it says as a todo in the code, we should probably make constants for the magic numbers
in the memory estimates - and you might want to come up with a better one for the deletioninfo


was (Author: graham sanderson):
Ok, added 7546.20_4.txt patch against 2.0 (with a bit of static debug stuff we probably don't
need)

Currently it is set to 10M + 7.5M/s just because it is cleaner with a power of 2 clock divider...
We could tweak these, but this worked well in my tests both on my laptop and on one of our
production class machines. It basically did exactly as you'd expect which was to kick in once
memory was actually being wasted at a reasonable rate.

A few random thoughts

# It would be nice to expose a metric on the CF (I'm not sure whether it should be just total
counts, or some fraction)... it'd need to be dealt with maybe in {{Memtable.FlushRunnable}}...
I left a package protected method {{usePessimisticLocking}}. I need your advice on whether/how
to expose this, but it'd be nice to watch in test (possibly with the actualy synchronization
turned off) to see how much it is triggered
# Note that we are actually running with {{concurrent_writes = 256}} in production as we have
lots of cores and very fast IO... so it seems likely that we can keep a lot of cores contended
in the hinting case... At a minimum of the rule of thumb would have put it at 128 which would
still be plenty high - I just thought I'd point that out
# I got rid of the unsafe again since we don't need it. I made a synchronized method rather
than plopping some synchronized blocks inside the method bodies because c2 compiler used to
be a bit reticent to inline methods with monitor usage and so it is kind of a habit for now
(also in this case it is probably just as clean if not cleaner).

> AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
> -----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7546
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: graham sanderson
>            Assignee: graham sanderson
>         Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_alt.txt,
suggestion1.txt, suggestion1_21.txt
>
>
> In order to preserve atomicity, this code attempts to read, clone/update, then CAS the
state of the partition.
> Under heavy contention for updating a single partition this can cause some fairly staggering
memory growth (the more cores on your machine the worst it gets).
> Whilst many usage patterns don't do highly concurrent updates to the same partition,
hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory
allocation rates can be seen (especially when the updates being hinted are small updates to
different partitions which can happen very fast on their own) - see CASSANDRA-7545
> It would be best to eliminate/reduce/limit the spinning memory allocation whilst not
slowing down the very common un-contended case.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message