cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Capwell (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-16144) TLS connections to the storage port on a node without server encryption configured causes java.io.IOException accessing missing keystore
Date Tue, 27 Oct 2020 20:15:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17221716#comment-17221716
] 

David Capwell edited comment on CASSANDRA-16144 at 10/27/20, 8:14 PM:
----------------------------------------------------------------------

Starting commit

CI Results (pending):
||Branch||Source||Circle CI||Jenkins||
|cassandra-2.2 |[branch|https://github.com/dcapwell/cassandra/tree/commit_remote_branch/CASSANDRA-16144-cassandra-2.2-6298EA10-5C5B-41A3-9575-78AA3A5A5249]
|[build|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=commit_remote_branch%2FCASSANDRA-16144-cassandra-2.2-6298EA10-5C5B-41A3-9575-78AA3A5A5249]
|[build|https://ci-cassandra.apache.org/job/Cassandra-devbranch/140/]|
|cassandra-3.0 |[branch|https://github.com/dcapwell/cassandra/tree/commit_remote_branch/CASSANDRA-16144-cassandra-3.0-6298EA10-5C5B-41A3-9575-78AA3A5A5249]
|[build|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=commit_remote_branch%2FCASSANDRA-16144-cassandra-3.0-6298EA10-5C5B-41A3-9575-78AA3A5A5249]
|[build|https://ci-cassandra.apache.org/job/Cassandra-devbranch/141/]|
|cassandra-3.11 |[branch|https://github.com/dcapwell/cassandra/tree/commit_remote_branch/CASSANDRA-16144-cassandra-3.11-6298EA10-5C5B-41A3-9575-78AA3A5A5249]
|[build|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=commit_remote_branch%2FCASSANDRA-16144-cassandra-3.11-6298EA10-5C5B-41A3-9575-78AA3A5A5249]
|[build|https://ci-cassandra.apache.org/job/Cassandra-devbranch/142/]|
|trunk |[branch|https://github.com/dcapwell/cassandra/tree/commit_remote_branch/CASSANDRA-16144-trunk-6298EA10-5C5B-41A3-9575-78AA3A5A5249]
|[build|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=commit_remote_branch%2FCASSANDRA-16144-trunk-6298EA10-5C5B-41A3-9575-78AA3A5A5249]
|[build|https://ci-cassandra.apache.org/job/Cassandra-devbranch/143/]|



was (Author: dcapwell):
Starting commit

CI Results (pending):
||Branch||Source||Circle CI||Jenkins||
|cassandra-2.2 |https://github.com/dcapwell/cassandra/tree/commit_remote_branch/CASSANDRA-16144-cassandra-2.2-6298EA10-5C5B-41A3-9575-78AA3A5A5249
|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=commit_remote_branch%2FCASSANDRA-16144-cassandra-2.2-6298EA10-5C5B-41A3-9575-78AA3A5A5249
|https://ci-cassandra.apache.org/job/Cassandra-devbranch/140/|
|cassandra-3.0 |https://github.com/dcapwell/cassandra/tree/commit_remote_branch/CASSANDRA-16144-cassandra-3.0-6298EA10-5C5B-41A3-9575-78AA3A5A5249
|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=commit_remote_branch%2FCASSANDRA-16144-cassandra-3.0-6298EA10-5C5B-41A3-9575-78AA3A5A5249
|https://ci-cassandra.apache.org/job/Cassandra-devbranch/141/|
|cassandra-3.11 |https://github.com/dcapwell/cassandra/tree/commit_remote_branch/CASSANDRA-16144-cassandra-3.11-6298EA10-5C5B-41A3-9575-78AA3A5A5249
|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=commit_remote_branch%2FCASSANDRA-16144-cassandra-3.11-6298EA10-5C5B-41A3-9575-78AA3A5A5249
|https://ci-cassandra.apache.org/job/Cassandra-devbranch/142/|
|trunk |https://github.com/dcapwell/cassandra/tree/commit_remote_branch/CASSANDRA-16144-trunk-6298EA10-5C5B-41A3-9575-78AA3A5A5249
|https://app.circleci.com/pipelines/github/dcapwell/cassandra?branch=commit_remote_branch%2FCASSANDRA-16144-trunk-6298EA10-5C5B-41A3-9575-78AA3A5A5249
|https://ci-cassandra.apache.org/job/Cassandra-devbranch/143/|


> TLS connections to the storage port on a node without server encryption configured causes
java.io.IOException accessing missing keystore
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-16144
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-16144
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Messaging/Internode
>            Reporter: Jon Meredith
>            Assignee: Jon Meredith
>            Priority: Normal
>             Fix For: 4.0-beta
>
>          Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> If a TLS connection is requested against a node with all encryption disabled by configuration,
> configured with
> {code}
> server_encryption_options: {optional:false, internode_encryption: none}
> {code}
> it logs the following error if no keystore exists for the node.
> {code}
> INFO  [Messaging-EventLoop-3-3] 2020-09-15T14:30:02,952 : - 127.0.0.1:7000->127.0.1.1:7000-URGENT_MESSAGES-[no-channel]
failed to connect
> io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: local1-i1/127.0.1.1:7000
> Caused by: java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
>        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
>        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
~[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
~[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
~[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
~[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
~[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at java.lang.Thread.run(Thread.java:834) [?:?]
> WARN  [Messaging-EventLoop-3-9] 2020-09-15T14:30:06,375 : - Failed to initialize a channel.
Closing: [id: 0x0746c157, L:/127.0.0.1:7000 - R:/127.0.0.1:59623]
> java.io.IOException: failed to build trust manager store for secure connections
>        at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:232)
~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
>        at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:300)
~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
>        at org.apache.cassandra.security.SSLFactory.getOrCreateSslContext(SSLFactory.java:276)
~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
>        at org.apache.cassandra.security.SSLFactory.getOrCreateSslContext(SSLFactory.java:257)
~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
>        at org.apache.cassandra.net.InboundConnectionInitiator$Initializer.initChannel(InboundConnectionInitiator.java:107)
~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
>        at org.apache.cassandra.net.InboundConnectionInitiator$Initializer.initChannel(InboundConnectionInitiator.java:71)
~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
>        at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:502)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[netty-all-4.1.50.Final.jar:4.1.50.Final]
>        at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: java.nio.file.NoSuchFileException: conf/.keystore
>        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
>        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
>        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
>        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
~[?:?]
>        at java.nio.file.Files.newByteChannel(Files.java:370) ~[?:?]
>        at java.nio.file.Files.newByteChannel(Files.java:421) ~[?:?]
>        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
~[?:?]
>        at java.nio.file.Files.newInputStream(Files.java:155) ~[?:?]
>        at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:207)
~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
>        ... 23 more
> {code}
> This happens regardless of the settings of server_encryption_options.optional, as the
OptionalSslHandler is installed if optional is true, and if optional is not true, an SSL handler
is always installed.
> https://github.com/apache/cassandra/blob/674b6cc1a5e905a9c234c649adaad2de79cfa560/src/java/org/apache/cassandra/net/InboundConnectionInitiator.java#L101
> CASSANDRA-15262 improved backward 4.0 backward compatability with 3.11 by defaulting
to allowing optional encrypted connections if optional was not explicitly configured, however
if an operator has not installed a keystore and an secure connection cannot possibly be established,
neither the optional or required SSL  handlers should be installed.
> Similarly if the operator has explicitly disabled SSL, then neither SSL handlers should
be installed.
> Temporarily commenting out both handlers and testing a TLS connection logs, an incoming
TLS connection logs an ERROR
> {code}
> ERROR [Messaging-EventLoop-3-1] 2020-09-15 15:09:48,898 InboundConnectionInitiator.java:355
- Failed to properly handshake with peer /127.0.0.1:60525. Closing the channel.
> io.netty.handler.codec.DecoderException: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic:
Read 369295616, Expected -900387334
>        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
>        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
>        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
>        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
>        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
>        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
>        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
>        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
>        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>        at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read 369295616,
Expected -900387334
>        at org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:333)
>        at org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167)
>        at org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:255)
>        at org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:248)
>        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
>        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
>        ... 17 common frames omitted
> {code}
> I think it would be helpful to an include a {{DisabledSslHandler}} that uses a similar
method to the {{OptionalSSLHandler}} of waiting for enough bytes to be received, and then
logging a warning about an SSL connection attempt on the port to help the operator spot misconfigurations
or security scans.
> There are another couple of related, but minor additional issues.
> The listening message does not currently indicate whether optional secure connections
are permitted, it should be explicit about whether secure connections are required, optional
and possibly include the internode mode.
> {code}
>  INFO  [main] 2020-09-15 15:09:33,045 InboundConnectionInitiator.java:128 - Listening
on address: (localhost/127.0.0.1:7000), nic: lo0, encryption: disabled
> {code}
> Also {{org.apache.cassandra.security.SSLFactory#buildKeyManagerFactory}} throws {{throw
new IOException("failed to build trust manager store for secure connections", e);}} when it
should be {{throw new IOException("failed to build key manager store for secure connections",
e);}} (probably a copy/paste from {{buildTrustManagerFactory}})
> Proposed changes to resolve this issue and improve behavior:
> * Introduce a DisabledSslHandler that logs a WARN if an SSL connection is attempted when
administratively disabled.
> * Add a check that the keystore exists before defaulting optional to false (as a path
to a keystore is hard-coded and simply not supplying it does not work)
> * Update the listening message to be explicit about what types of encrypted connections
will be accepted.
> * Fix the Keystone IOException to correctly identify the Keystore as the culprit



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message