cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-10994) Move away from SEDA to TPC, stage 1
Date Mon, 11 Jan 2016 19:41:39 GMT


Ariel Weisberg commented on CASSANDRA-10994:

One of the big benefits of TPC is small tasks where the overhead of context switching to a
task is large.

For large tasks the overhead of relying on a thread scheduler and real threads is in the noise.
For tasks like compaction that can operate against immutable data sets there isn't a lot of
overhead in terms of complexity in continuing to use a separate thread to access the data.
Scheduling decisions and core affinity that you can express with TPC can also be expressed
with real threads. The binding requires OS support, but scheduling is something you can have
decent control of without OS support.

I think that TPC delivers the most value for request processing where the data for a request
is already available in memory (all writes, some reads). It also delivers a lot of value if
you can feasibly schedule hundreds of thousands of disk IO requests asynchronously, but that
seems like a hard thing to make happen from Java.

Another thing I want to advocate for is not assuming that for cores that are provisioned to
run a single thread that all threads are completely uniform in their task distribution. There
is research that demonstrates that narrowing the focus of tasks assigned to a core can yield
better performance even when you take into account extra messaging between cores. 

This is important for networking and file IO which both regularly enter the kernel. It's already
proposed to give file IO dedicated threads (possibly cores), and I think it's a good idea
to test with dedicated vs separate cores for network IO.

My biggest concern with TPC is load skew especially temporal skew. If you have a completely
uniform task distribution (no balancing) and no work stealing then a big compaction kicking
off on one core is going to cut its capacity by some substantial percentage. Eventually all
outstanding requests will be for that slow core. You can replace core with node and it's basically
the same problem.

> Move away from SEDA to TPC, stage 1
> -----------------------------------
>                 Key: CASSANDRA-10994
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Aleksey Yeschenko
> To start off the transition, I propose a modest (if not underwhelming) set of changes
for stage 1:
> 1. Convert read and write request paths to be fully non-blocking, and execute them directly
within Netty context, avoiding any thread handoff (CASSANDRA-10993)
> 2. Implement our own in-process page cache to complement (1) (CASSANDRA-5863)
> (2) is necessary to enable serving reads for memory-resident rows without handing them
off to another stage.
> However, read requests that cannot be served from the cache will have to be handed off
to a new thread pool (replacing the old {{READ}} stage), that would execute individual {{ReadCommand}}
s using blocking I/O.
> The extra thread pool here is unfortunate, but cannot be avoided, as we have to support
filesystems that aren’t xfs.
> For stage 1, we are not going to partition data ownership yet - every worker thread will
be able to serve requests for any token. We are also not going to introduce processor affinity,
or alter our partition or memtable data structures.
> Memtable flushing, compaction, and repair will not be modified beyond necessary changes
caused by CASSANDRA-5863.
> With (1) and (2) combined we expect to see noticeable improvements for at least {{CL.ONE}}
reads that can be served from memory and RF=1 writes. That, and not introducing any noticeable
performance regressions for other types of requests is the success criteria for stage 1.
> I should note that we *could* do more transition work in parallel - in particular have
the team working on making other components non-blocking, but don’t want to go that way
for the following reasons:
> - Cassandra is a solid production-ready database, and should remain so. Introducing too
much change in big chunks would make it hard to maintain stability
> - there is an argument to be made regarding not having (some of) maintenance task share
the event loop with read and write requests handling loops, as they don’t necessarily benefit
from it (cc [~aweisberg], who has an expanded comment prepared on this). Once we are done
with stage 1, we will evaluate whether or not we should do that
> - introducing change progressively would give projects built on Cassandra (Stratio lucene-based
search, Tuplejump’s integration, and DSE) to catch up and make necessary changes as they
are being introduced
> This ticket will serve as an umbrella issue for all the work necessary for stage 1.

This message was sent by Atlassian JIRA

View raw message