cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Schuller (JIRA)" <>
Subject [jira] Commented: (CASSANDRA-1657) support in-memory column families
Date Wed, 02 Feb 2011 08:01:29 GMT


Peter Schuller commented on CASSANDRA-1657:

Depending on use-case maybe, but I don't think always. Primarily for three reasons:

(1) The row cache is not at all as efficient in terms of memory use. Assuming the premise
that you really want it to be guaranteed in memory is true (rather than a 10% active set),
the on-disk format is considerably more compact (more so for very small rows with very small
columns, than for medium-sized cf:s with less keys). A nice bonus is avoiding GC pressure,
and edge cases like the odd large row and difficulty sizing the row cache.

(2) The row-cache has a sever trade-off in terms of start-up time due to it being seek-bound
(although for CF:s that are supposed to be in-memory anyway it may be mitigated by pre-heating
it by streaming through it; but that feels more like a hack). A kernel-driven sequential pre-population
is vastly preferred ;)

(3) Hopefully with future improvements in compaction (e.g. limiting maximum sstable size so
that compaction can happen to reasonably sized chunks, thus not doubling memory use) an in-memory
CF would stay in-memory completely during compaction (only incurring deferred writes). In
other words, it would contribute significantly less to (disk i/o) compaction overhead than
if the active set was in row cache.

I do assume that the use-case is such that optimizing for avoiding serialization/deserialization
is not the issue here, and that the importance of keeping the data off disk is more important
than squeezing the last bit of CPU efficiency out of it.

Since Cassandra is going the route of native code and embracing things like posix_fadvise/mincore/mlock
etc anyway, it seemed like a reasonable thing to support. However of course if other features
end up catering sufficiently to the same goal, the suggestion may become obsolete.

Also, I still like the stuff in CASSANDRA-1625 (specifically
which overlaps with this.

> support in-memory column families
> ---------------------------------
>                 Key: CASSANDRA-1657
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Peter Schuller
>            Priority: Minor
> Some workloads are such that you absolutely depend on column families being in-memory
for performance, yet you most definitely want all the things that Cassandra offers in terms
of replication, consistency, durability etc.
> In order to semi-deterministically ensure acceptable performance for such data, Cassandra
could support in-memory column families. Such an in-memory column family would imply that
mlock() be used on sstables for this column family. On start-up and on compaction completion,
they could be mmap():ed with MAP_POPULATE (Linux specific) or else just mmap():ed + mlock():ed
in such a way as to otherwise guarantee it is in-memory (such as userland traversal of the
entire file).

This message is automatically generated by JIRA.
For more information on JIRA, see:


View raw message