cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-6689) Partially Off Heap Memtables
Date Tue, 04 Mar 2014 10:33:22 GMT


Benedict commented on CASSANDRA-6689:

bq. What I want is to see more control over how we deal with cleaner threads, as the main
point to be able to run with smaller heaps, and thread stacks could consume ~300K we are risking
to OOM in certain situations because we don't control whole thread life time, that would also
help for people in environments guided by ulimit or similar.

Just for clarity, it's the _collector_ threads, not the _cleaner_ threads you're worried about,
right? Because there is only ever one cleaner thread. Like I say, this is mostly trying to
help with an anti-pattern, so I'm not sure it's worth addressing too aggressively in this
ticket. In real terms you'll struggle to spin up very many threads at all, but let's put an
arbitrary limit of 100 threads on the pool for now. I don't think this is worth offering configuration
options to the user here.

bq. Please do think about it, we need to try to reduce the complexity of this to minimize
bus factor

As I said, I (still) can't see a good way to use this information to simplify matters usefully.
If we flattened Pool, AllocatorGroup and Allocator all into the one object, and we had a per-CF
memory limit, we would be able to simplify all of the SubPool and SubAllocator memory management
considerably, but then we have the issue that there's no global management of memory resources,
nor shared re-use. I don't think this would simplify the actual recycling of memory regions
that is introduced in this ticket at all though, but it's possible I'm missing something.

Suggestions always welcome :-)

bq. Exactly, maybe it's time to re-evaluate trade-off between copying data to per-allocated
regions with limited ttl vs. tracking data all around the place.

This is the opposite suggestion, right? :-)   It is definitely something worth exploring in

bq. Have you had a chance to do more on vs. off heap performance testing?

I'm not sure Marcus was performance testing. Personally I'm most interested in correctness
right now: performance should follow (in particular, slow flushing is certainly a "bug" which
can be fixed). Certainly for overwrite workloads I've demonstrated the reduced flushing on
my box, but further testing of the impact is likely to be a long drawn out process, and probably
beyond scope of this ticket IMO.

Performance testing scope might include:
- Establish how much heap limit reduction this permits
- Establish a severe GC pause inducing workload to see if this mitigates
- Establish if G1GC is made viable with this, and if not, what remaining impacts there are
to adoption that might be lifted
- Establish how much improvement to overwrite workloads (or mixed workloads, where reduced
overwrite workload memory consumption => reduced non-overwrite workload flushing)
- Establish new reasonable upper memory limit for heap and memtable (both independently),
for different workloads. e.g. large data payload workloads may now practicably support much
larger memtables and dramatically reduced flushing
- Establish impact on upper *data* limit for different workloads, with same memory limit as
olde, and the impact on size-of-flush and concomitant reduction in write amplification

> Partially Off Heap Memtables
> ----------------------------
>                 Key: CASSANDRA-6689
>                 URL:
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Benedict
>            Assignee: Benedict
>             Fix For: 2.1 beta2
>         Attachments: CASSANDRA-6689-small-changes.patch
> Move the contents of ByteBuffers off-heap for records written to a memtable.
> (See comments for details)

This message was sent by Atlassian JIRA

View raw message