cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Brown (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-8457) nio MessagingService
Date Fri, 08 Jul 2016 22:32:11 GMT


Jason Brown commented on CASSANDRA-8457:

Here's the first pass at switching internode messaging to netty.

I've tried to preserve all of the functionality/behaviors of the existing implementation as
I could, and some aspects were a bit tricky in the non-blocking IO, netty world. I've also
extensively documented the code as much as I can, and I still want to add more high-level
docs on 1) the internode protocol itself, and 2) the use of netty in internode messaging.
Hopefully the current state of documentation helps understanding and reviewing the changes.
Here's some high-level notes as points of departure/interest/discussion:

- I've left the existing {{OutboundTcpConnection}} code largely intact for the short term
(read on for more detail). But mostly the new and existing behaviors and coexist in the code
together (though not at run time)
- There is a yaml property to enable/disable using netty for internode messaging. If disabled,
we'll fall back to the existing {{OutboundTcpConnection}} code. Part of this stems from the
fact that streaming also uses the same the socket infrastructure as internode messaging handshake
as messaging, and streaming would be broken without the {{OutboundTcpConnection}} implementation.
I am knees deep in switching streaming over to a non-blocking, netty-based solution, but that
is a separate ticket/body of work.
- In order to support non-blocking IO, I've altered the internode messaging protocol such
that each message is framed, and the frame contains a message size. The protocol change is
what forces these changes to happen at a major rev update, hence 4.0
- Backward compatibility - We will need to handle the case of cluster upgrade where some nodes
are on the previous version of the protocol (not upgraded), and some are upgraded. The upgraded
nodes will still need to behave and operate correctly with the older nodes, and that functionality
is encapsulated and documented in {{LegacyClientHandler}} (for the receive side) and {{MessageOutHandler}}
for the send side.
- Message coalescing - The existing behaviors in {{CoalescingStrategies}} are predicated on
parking the thread to allow outbound messages to arrive (and be coalesced). Parking a thread
in a non-blocking/netty context is a bad thing, so I've inverted the behavior of message coalescing
a bit. Instead of blocking a thread, I've extended the {{CoalescingStrategies.CoalescingStrategy}}
implementations to return a 'time to wait' to left messages arrive for sending. I then schedule
a task in the netty scheduler to execute that many nanoseconds in the future, queuing up incoming
message, and then send them out when the scheduled task executes (this is {{CoalescingMessageOutHandler}}).
I've also added callback functions to {{CoalescingStrategies.CoalescingStrategy}} implementations
for the non-blocking paradigm to record updates to the strategy (for recalculation of the
time window, etc).
- Message flushing - Currently in {{OutboundTcpConnection}}, we only call flush on the output
stream if the backlog is empty (there's no more messages to send to the peer). Unfortunately
there's no equivalent API in netty to know there's any messages in the channel waiting to
be sent. The solution that I've gone with is to have a shared counter outside of the channel
{{InternodeMessagingConnection#outboundCount}} and inside the channel {{CoalescingMessageOutHandler#outboundCounter}},
and when {{CoalescingMessageOutHandler}} sees the value of that counter is zero, it knows
it can explicitly call flush. I'm not entirely thrilled with this approach, and there's some
potential race/correctness problems (and complexity!) when reconnections occur, so I'm open
to suggestions on how to achieve this functionality.
- I've included support for netty's OpenSSL library. The operator will need to deploy an extra
netty jar ( to get the OpenSSL behavior (I'm
not sure if we can or want to include it in our distro). {{SSLFactory}} needed to be refactored
a bit to support the OpenSSL functionality.

I'll be doing some more extensive testing next week (including a more thorough exploration
of the backward compatibility).

> nio MessagingService
> --------------------
>                 Key: CASSANDRA-8457
>                 URL:
>             Project: Cassandra
>          Issue Type: New Feature
>            Reporter: Jonathan Ellis
>            Priority: Minor
>              Labels: netty, performance
>             Fix For: 4.x
> Thread-per-peer (actually two each incoming and outbound) is a big contributor to context
switching, especially for larger clusters.  Let's look at switching to nio, possibly via Netty.

This message was sent by Atlassian JIRA

View raw message