cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-6689) Partially Off Heap Memtables
Date Wed, 05 Mar 2014 02:27:43 GMT


Pavel Yaskevich commented on CASSANDRA-6689:

bq. Zero copy would certainly be nice to retain. Otherwise why not just ref-count the memory?

I'm not sure what is to retain here if we do that copy when we send to the wire. Wouldn't
we better of with ref count and copy but having separate allocator for temporary things (e.g.
write/read buffers) and for more long term (e.g. memtable, compression buffers)?

So can you please answer the question - what is the biggest advantage of tracking everything,
introducing separate gc, sort of RCU (i'm looking at you OpOrder) vs. mem copy + different
allocators for different things if there is a fixed size pre-allocated buffer pool off heap
(we can also do COW for some of the chunks)? Copy is *not* be a big problem (regarding throughput/latency)
comparing to other operations (do a simple memcpy test and see how much mb/s can you get from
copying from one pre-allocated pool to another), there is a trade-off of course as we are
going to spend more memory to store temporary things but as we have a fixed number of threads
it is going to work out the same way as for buffering open files in the steady system state...
Temporary memory allocated by readers is *exactly* what we should be managing at the first
place because they allocate the most and it always the biggest concern for us (ParNew pauses
are reaching 300 ms in 1 sec. intervals), that's why I'm saying let's use "low level" allocator
for things like reads with it's own arena and chunk sizes, as we originally wanted and have
a separate "low level" allocator for the memtable, preallocate at the startup, return buffers
to the pool when we serialize for the wire and go from there with global thresholds etc.

After all that said, I would suggest that we do a separate "low level" allocator in this ticket
(the same way as jemalloc) and plug it into the memtable and do a copy on read path, once
that step is done we can plug-in allocator for all of the serialization/deserialization logic
that we have when it's going from/to the wire and the last step would be to use allocator
for all of the sstable read operations.

All this might seem irrelevant for some people but I can say that after fire-fighting Cassandra
for some time for different use cases, it's not the memtable which creates the most of the
noise and memory presure in the system (even tho it uses big chunk of heap) but the reads
and internode communication (especially the latter).

> Partially Off Heap Memtables
> ----------------------------
>                 Key: CASSANDRA-6689
>                 URL:
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Benedict
>            Assignee: Benedict
>             Fix For: 2.1 beta2
>         Attachments: CASSANDRA-6689-small-changes.patch
> Move the contents of ByteBuffers off-heap for records written to a memtable.
> (See comments for details)

This message was sent by Atlassian JIRA

View raw message