cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11854) Remove finished streaming connections from MessagingService
Date Mon, 27 Jun 2016 17:16:52 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351418#comment-15351418
] 

Paulo Motta commented on CASSANDRA-11854:
-----------------------------------------

bq. In SocketThread#close() we iterate over the connections and close each one, this will
not work now since the connections will be empty, right? 

Actually this would only be empty if there are no ongoing stream sessions, otherwise active
stream sessions would only be cleared after the {{StreamResultFuture}} was completed.

bq. Should we perhaps close() the socket when we remove it from group? They were only closed
on server shutdown before so maybe we don't need to do that?

It's not safe to close the socket when a stream session is finished (failed or completed),
because the outgoing stream handler may still need to send queued {{SessionComplete}} or {{SessionFailed}}
messages after the {{StreamResultFuture}} returns. For this reason, there may be edge cases
with the previous approach when there are still active streaming connections not closed by
{{MessagingService.shutdown()}}.

In order to guarantee streaming sockets are only removed from {{SocketThread.connections}}
after the {{(Incoming|Outgoing)MessageHandler}} are finished, the new approach attaches the
{{IncomingStreamingConnection}} to the {{ConnectionHandler}}, so on {{ConnectionHandler.signalCloseDone()}}
the {{IncomingStreamingConnection}} is closed, closing the socket and removing it from {{SocketThread.connections}}.
I also added a unit test to verify that all sockets are cleared from {{SocketThread.connections}}
after the stream session finishes.

New patch and tests below:

||2.1||2.2||3.0||trunk||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.1...pauloricardomg:2.1-11854-v3]|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:2.2-11854-v3]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...pauloricardomg:3.0-11854-v3]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-11854-v3]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-11854-v3-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-11854-v3-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-11854-v3-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-11854-v3-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-11854-v3-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-11854-v3-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-11854-v3-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-11854-v3-dtest/lastCompletedBuild/testReport/]|

commit info: minor conflicts all the way up to trunk :(

> Remove finished streaming connections from MessagingService
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-11854
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11854
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Paulo Motta
>            Assignee: Paulo Motta
>         Attachments: oom.png
>
>
> When a new {{IncomingStreamingConnection}} is created, [we register it in the connections
map|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/MessagingService.java#L1109]
of {{MessagingService}}, but we [only remove it if there is an exception|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/IncomingStreamingConnection.java#L83]
while attaching the socket to the stream session.
> On nodes with SSL and large number of vnodes, after many repair sessions these old connections
can accumulate and cause OOM (heap dump attached).
> The connection should be removed from the connections map after if it's finished in order
to be garbage collected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message