cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-8789) Revisit how OutboundTcpConnection pools two connections for different message types
Date Fri, 13 Feb 2015 18:09:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320467#comment-14320467
] 

Ariel Weisberg edited comment on CASSANDRA-8789 at 2/13/15 6:08 PM:
--------------------------------------------------------------------

I tried using a single socket it is a big improvement at medium concurrency. At high concurrency
there is no improvement because it is already CPU bound, but at 100 threads it goes from 140k
to 225k. This is with the same workload I have been running to test coalescing which is small
keys and small values 100% read CL = ALL, 100k row dataset.

Part of this is that at 140k with a coalescing window of 200 microseconds coalescing doesn't
occur even though coalescing would actually shrink the gap under the window. The extra traffic
triggers coalescing at the 200 microsecond level. The previous number when time horizon mistakenly
coalesced was 180k so this is an improvement on top of that as well.

One finicky thing is that we don't actually know the serialized size of messages and calculating
that isn't necessarily cheap. It requires walking the entire object graph for the message,
and then repeating that process for serialization. For simple messages it might not matter,
for large ones it might be dwarfed by everything else. I will measure the cost for small messages.

One other question, is 64k a useful threshold? Will systems regularly hit that threshold as
part of operations like repair, streaming, or something else I don't know about?



was (Author: aweisberg):
I tried using a single socket it is a big improvement at medium concurrency. At high concurrency
there is no improvement because it is already CPU bound, but at 100 threads it goes from 140k
to 225k. This is with the same workload I have been running to test coalescing which is small
keys and small values 100% read CL = ALL, 100k row dataset.

Part of this is that at 140k with a coalescing window of 200 microseconds coalescing doesn't
occur even though coalescing would actually shrink the gap under the window. The extra traffic
triggers coalescing at the 200 microsecond level. The previous number when time horizon mistakenly
coalesced was 180k so this is an improvement on top of that as well.

One finicky thing is that we don't actually know the serialized size of messages and calculating
that isn't necessarily cheap. It requires walking the entire object graph for the message,
and then repeating that process for serialization. For simple messages it might not matter,
for large ones it might be dwarfed by everything else. I will measure the cost for small messages.



> Revisit how OutboundTcpConnection pools two connections for different message types
> -----------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-8789
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8789
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 3.0
>
>
> I was looking at this trying to understand what messages flow over which connection.
> For reads the request goes out over the command connection and the response comes back
over the ack connection.
> For writes the request goes out over the command connection and the response comes back
over the command connection.
> Reads get a dedicated socket for responses. Mutation commands and responses both travel
over the same socket along with read requests.
> Sockets are used uni-directional so there are actually four sockets in play and four
threads at each node (2 inbounded, 2 outbound).
> CASSANDRA-488 doesn't leave a record what the impact of this change was. If someone remembers
what situations were made better it would be good to know.
> I am not clear on when/how this is helpful. The consumer side shouldn't be blocking so
the only head of line blocking issue is the time it takes to transfer data over the wire.
> If message size is the cause of blocking issues then the current design mixes small messages
and large messages on the same connection retaining the head of line blocking.
> Read requests share the same connection as write requests (which are large), and write
acknowledgments (which are small) share the same connections as write requests. The only winner
is read acknowledgements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message