cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-9237) Gossip messages subject to head of line blocking by other intra-cluster traffic
Date Tue, 28 Apr 2015 16:06:06 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517288#comment-14517288
] 

Benedict edited comment on CASSANDRA-9237 at 4/28/15 4:05 PM:
--------------------------------------------------------------

TCP connections are generally designed to share bandwidth fairly, so having an extra connection
means a low traffic high priority connection will be more likely to see its messages arrive
promptly. Either way, having multiple connections means that if one gets backed up you can
at least hop to the front of the application level queues that are now backed up.

I agree that extra threads are unacceptable, but having a single thread serving a collection
of NIO sockets to each host for Gossip, as I suggested in CASSANDRA-8789 seems like it has
no major downsides besides the work involved. Ideally we would precede such work by isolating
the transport implementation of MessagingService from its utilisation (and permit different
messages go over different transports), so that we can drop in a multitude of different schemes
(OIO, NIO, UDP, SCTP, whatever), and then the problem becomes very tractable. I would argue
it becomes very sensible, even.

Constraining huge messages to such an NIO socket also makes sense to me, since I doubt the
throughput problems we see with NIO are a major issue for large messages, and we don't really
want lots of unnecessary threads hanging around. This also makes it easier to transparently
switch implementation based on e.g. cluster size, or frequency of communication (non-vnode
clusters could have OIO connections to their direct peers, and NIO to everyone else, for instance).


was (Author: benedict):
TCP connections are generally designed to share bandwidth fairly, so having an extra connection
means a low traffic high priority connection will be more likely to see its messages arrive
promptly. Either way, having multiple connections means that if one gets backed up you can
at least hop to the front of the application level queues that are now backed up.

I agree that extra threads are unacceptable, but having a single thread serving a collection
of NIO sockets to each host for Gossip, as I suggested in CASSANDRA-8789 seems like it has
no major downsides besides the work involved. Ideally we would precede such work by isolating
the transport implementation of MessagingService from its utilisation (and permit different
messages go over different transports), so that we can drop in a multitude of different schemes
(OIO, NIO, UDP, SCTP, whatever), and then the problem becomes very tractable. I would argue
it becomes very sensible, even.

> Gossip messages subject to head of line blocking by other intra-cluster traffic
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-9237
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9237
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>
> Reported as an issue over less than perfect networks like VPNs between data centers.
> Gossip goes over the small message socket where small is 64k which isn't particularly
small. This is done for performance to keep most traffic on one hot socket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message