cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5239) Asynchronous (non-blocking) StorageProxy
Date Wed, 23 Jul 2014 21:14:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072351#comment-14072351
] 

Benedict commented on CASSANDRA-5239:
-------------------------------------

The stated goal of this ticket is to reduce context switching costs - this is only a practical
concern for local-only in-memory requests. Any impact of context switching costs off the wire
from a remote node are unlikely to be impacted by this patch, as they can only really practically
cause problems through bottlenecking the rate of message processing off the wire, but that
is another context switch removed from this one. This might be helped by introducing SharedExecutorPool
to the INTERNAL_RESPONSE stage, but that should be benchmarked separately to see if it makes
a difference. It was an oversight in CASSANDRA-4718 to not do this.

As far as local-only in-memory requests go, SharedExecutorPool has already reduced the context
switch cost to minimal so long as RPC count <= max_concurrent_reads, by stealing a work
permit and executing it inline; per-disk access coordination will reduce it to always minimal
by eliminating the extraneous (read stage) thread pool, so there will be no context switches
at all.

That said, there *is* a context switch cost still there that we could eliminate, but I'm not
particularly convinced it warrants removal. 

As far as streaming the pipeline is concerned, I'm +100 on the goal, but it's not clear to
me that this makes that easier or harder as it currently stands. It may well benefit from
being asynchronous, but a future based design doesn't seem to easily support the incremental
delivery/processing of results.



> Asynchronous (non-blocking) StorageProxy
> ----------------------------------------
>
>                 Key: CASSANDRA-5239
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5239
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 2.0 beta 1
>            Reporter: Vijay
>            Assignee: Sylvain Lebresne
>              Labels: performance
>             Fix For: 3.0
>
>
> Problem Statement: 
> Currently we have "rpc_min_threads, rpc_max_threads"/ "native_transport_min_threads/native_transport_max_threads"
all of the threads in the TPE are blocking and takes resources, the threads are mostly sleeping.
Increasing the Context switch costs.
> Details: 
> We should change StorageProxy methods to provide a callback which contains the location
where the results has to be written. When the response arrive StorageProxy callback can write
the results directly into the connection. Timeouts can be handled in the same way.
> Fixing Netty should be trivial with some refactor in the storage proxy (currently it
is one method call for sending the request and waiting) we need callback.
> Fixing Thrift may be harder because thrift calls the method and expects a return value.
We might need to write a custom Codec on Netty for thrift support, which can potentially do
callbacks (A Custom codec may be similar to http://engineering.twitter.com/2011/04/twitter-search-is-now-3x-faster_1656.html
but we dont know details about it). Another option is to update thrift to have a callback.
> FYI, The motivation for this ticket is from another project which i am working on with
similar Proxy (blocking Netty transport) and making it Async gave us 2x throughput improvement.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message