cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
Date Wed, 15 Feb 2017 14:48:41 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Paulo Motta updated CASSANDRA-12886:
------------------------------------
    Status: Patch Available  (was: Open)

This is hard to reproduce but I've got a similar issue when working on CASSANDRA-11841, and
what is probably happening is a race where the handler sender thread is started and pushes
a message to the socket before the init message is sent, what breaks the connection in the
receiver side. In order to avoid this, we must first send the init message before starting
the handler thread. This should already be fixed on 3.10 by CASSANDRA-11841, but this fix
is for 2.2+.

||2.2||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:2.2-12886]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-12886-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-12886-dtest/lastCompletedBuild/testReport/]|

> Streaming failed due to SSL Socket connection reset
> ---------------------------------------------------
>
>                 Key: CASSANDRA-12886
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12886
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Bing Wu
>            Assignee: Paulo Motta
>         Attachments: debug.log.2016-11-10_2319.gz
>
>
> While running "nodetool repair", I see many instances of "javax.net.ssl.SSLException:
java.net.SocketException: Connection reset" in system.logs on some nodes in the cluster. Timestamps
correspond to streaming source/initiator's error messages of "sync failed between ..."
> Setup: 
> - Cassandra 3.7.01 
> - CentOS 6.7 in AWS (multi-region)
> - JDK version: {noformat}
> java version "1.8.0_102"
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
> {noformat}
> - cassandra.yaml:
> {noformat}
> server_encryption_options:
>     internode_encryption: all
>     keystore: [path]
>     keystore_password: [password]
>     truststore: [path]
>     truststore_password: [password]
>     # More advanced defaults below:
>     # protocol: TLS
>     # algorithm: SunX509
>     # store_type: JKS
>     # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
>     require_client_auth: false
> {noformat}
> Error messages in system.log on the target host:
> {noformat}
> ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475 StreamSession.java:529
- [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e] Streaming error occurred on session with
peer 54.247.111.232
> javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException:
java.net.SocketException: Connection reset
>         at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541) ~[na:1.8.0_102]
>         at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553) ~[na:1.8.0_102]
>         at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) ~[na:1.8.0_102]
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_102]
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_102]
>         at org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
~[apache-cassandra-3.7.0.jar:3.7.0]
>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
[apache-cassandra-3.7.0.jar:3.7.0]
>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
[apache-cassandra-3.7.0.jar:3.7.0]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message