From user-return-64397-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Tue Aug 27 00:20:07 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4EE21180645 for ; Tue, 27 Aug 2019 02:20:07 +0200 (CEST) Received: (qmail 78990 invoked by uid 500); 27 Aug 2019 00:20:03 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 78974 invoked by uid 99); 27 Aug 2019 00:20:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Aug 2019 00:20:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id C0F7AC0327 for ; Tue, 27 Aug 2019 00:20:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.9 X-Spam-Level: * X-Spam-Status: No, score=1.9 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, HTTPS_HTTP_MISMATCH=0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=salesforce.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id S9-y2nhTu84l for ; Tue, 27 Aug 2019 00:20:00 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.215.176; helo=mail-pg1-f176.google.com; envelope-from=mcarlise@salesforce.com; receiver= Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id C7A57BDEA4 for ; Tue, 27 Aug 2019 00:19:59 +0000 (UTC) Received: by mail-pg1-f176.google.com with SMTP id u17so11565466pgi.6 for ; Mon, 26 Aug 2019 17:19:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=OxmCfCscUdRhOrjLTOSnM7XrY40gnhxA+Ay5n9sPOZs=; b=IqP2O0m82gbdk9rZgt6rel0i/vbAZBnRFBSJNW6C8AGKLFN6R/tq3wPy+PIO+/wcYL oyIVkbE+qINLFklR6In8iNhvXsNhgkrtXY/Sy1V7bhMWq7uaL3LwPpDGikK1//RYOwb/ sEs/ftEiQ9mWXz61KhFESA4iiIfdgoUBZlIMwbcn27tgwDJDEYrfHFKEWvTYQMbP6022 Ntwy/fr0Z3C/mNNVZ9ChHVoY9ko3XGVn1Lok2+5N8ynkcyYLh4AhhA7EAlAH9HBybbll KAFcbPhFqVHDf++q13IyGVW7aONO03HdJRyPh0QMVyeX0u8OQSqeZb79oQ4aRY6ltAsI hg6g== X-Gm-Message-State: APjAAAWguyDhFcsXDYc5M2y8osxxKZkV/m0fs+gPtc7BstWXdYb4jRBa FfPFZLbD9iRHDJrrvYP565CHyjngd/IAFqwLDt1x8T3YbPY= X-Google-Smtp-Source: APXvYqxKdW5/WUZ+gOntEeYNiPynDTYQkw02xo5zZkg5BsCsXTpnMuDxjmnFAbTdJ8XnYdq3mQ+cPPH4qYkjOfS94A8= X-Received: by 2002:a63:9245:: with SMTP id s5mr19196573pgn.123.1566865198341; Mon, 26 Aug 2019 17:19:58 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Michael Carlise Date: Mon, 26 Aug 2019 20:19:22 -0400 Message-ID: Subject: Re: unable to gossip with peers exception when internode encryption is set to any setting other than 'none' To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="0000000000008f207805910e3b24" --0000000000008f207805910e3b24 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The version given by apt is 8u162-b12-1. Which I think corresponds to openJDK-8-162. When I run jrunscript -e 'print (javax.crypto.Cipher.getMaxAllowedKeyLength("RC5") >=3D 256);' the command returns true. Not sure if that is the best way to verify JCE installed. Michael Carlise On Mon, Aug 26, 2019 at 5:47 PM Marc Selwan wrote: > which exact version of OpenJDK are you using? Is it possible you don't > have JCE on those nodes? (I believe more recent versions of Java 8 has th= is > baked in so that might not be it) > > > *Marc Selwan | *DataStax *| *PM, Server Team *|* *(925) 413-7079* *|* > Twitter > > * Quick links | *DataStax *| *Training > *| *Documentation > > *| *Downloads > > > > On Mon, Aug 26, 2019 at 1:56 PM Michael Carlise > wrote: > >> >> I originally opened this issue on stackoverflow ( >> https://stackoverflow.com/questions/57516660/cassandra-node-to-node-encr= yption-throws-unable-to-gossip-with-peers-exception >> >> ). >> >> However, I haven't gotten any responses in over a week. I'm going to >> post it here and maybe someone will have an idea on where I can look. >> >> We currently run a multi region cassandra cluster in AWS. It runs in fou= r >> regions, 12 nodes per region. It runs without node to node encryption (o= r >> client encryption either). We are trying to enable inter datacenter node= to >> node encryption. However, when we flip encryption over we get an excepti= on >> that nodes are unable to gossip with any peers. >> >> It could possibly be that we didn't build our jks keystore/truststores >> correctly (more on how we built these files below). But, we additionally= do >> not see intra datacenter communication working (which should be set to >> unencrypted communication). Additionally, cqlsh cannot connect to the no= de >> either; even though we have (by default) client_auth_required set to >> false. >> >> ERROR [main] 2019-08-15 18:46:32,241 CassandraDaemon.java:749 - Exceptio= n encountered during startup >> java.lang.RuntimeException: Unable to gossip with any peers >> at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java= :1435) ~[apache-cassandra-3.11.4.jar:3.11.4] >> at org.apache.cassandra.service.StorageService.checkForEndpointC= ollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4] >> at org.apache.cassandra.service.StorageService.prepareToJoin(Sto= rageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4] >> at org.apache.cassandra.service.StorageService.initServer(Storag= eService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4] >> at org.apache.cassandra.service.StorageService.initServer(Storag= eService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4] >> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraD= aemon.java:388) [apache-cassandra-3.11.4.jar:3.11.4] >> at org.apache.cassandra.service.CassandraDaemon.activate(Cassand= raDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4] >> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDa= emon.java:732) [apache-cassandra-3.11.4.jar:3.11.4] >> INFO [main] 2019-08-15 18:47:07,384 YamlConfigurationLoader.java:89 - C= onfiguration location: file:/etc/cassandra/cassandra.yaml >> >> >> Something to note is that this error message occurs after a few minutes >> of the node being up. (i.e. there is a delay between start up before thi= s >> exception is thrown). >> >> *Information about our cassandra setup* >> >> cassandra version: 3.11.4 >> JDK version: openjdk-8. >> Linux: Ubuntu 18.04 (bionic). >> >> *cassandra.yaml* >> >> endpoint_snitch: Ec2MultiRegionSnitch >> >> server_encryption_options: >> internode_encryption: dc >> keystore: >> keystore_password: >> truststore: >> truststore_password: >> >> client_encryption_options: >> enabled: false >> >> *cassandra-rackdc.properties* >> >> prefer_local=3Dtrue >> >> *No obvious errors with SSH output* >> >> When starting cassandra with JVM_OPTS=3D"$JVM_OPTS -Djavax.net.debug=3Ds= sl" added >> to cassandra-env.sh we see SSL logs printed to stdout (*Note: Subject >> and Issuer were omitted on purpose)*. >> >> found key for : cassy-us-west-2 >> adding as trusted cert: >> Subject: ... >> Issuer: ... >> Algorithm: RSA; Serial number: 0xdad28d843fc73325d4c1a75207d4e74 >> Valid from Fri May 27 00:00:00 UTC 2016 until Tue May 26 23:59:59 UTC = 2026 >> >> ... >> >> trigger seeding of SecureRandom >> done seeding SecureRandom >> >> Looking at Java SE SSL/TLS connection debugging >> , >> this looks correct. But to note, we see this series of messages (along w= ith >> the RSA key signature output) repeated several times in rapid fire. We >> never observe any messages about the trust store being added; however th= at >> might be something that occurs only on client initiation (?) >> >> Additionally, we do see cassandra report that the Encrypted Messaging >> service has been started. >> >> INFO [main] 2019-08-15 18:45:31,022 MessagingService.java:704 - Startin= g Encrypted Messaging Service on SSL port 7001 >> >> *Doesn't appear to be a cassandra.yaml configuration problem* >> >> We can bring the node back online by simply configuring internode_encryp= tion: >> none. This action seems to rule out a broadcast_address or rpc_address >> configuration problem. >> >> *How we built our keystore/truststores* >> >> We followed the basic template datastax docs for preparing SSL >> certificates >> . >> One minor difference was that our private key and CSRs were generated us= ing >> openssl. One per each region (we plan to share key/signed certs across >> nodes in regions). This was created using a command template as: >> >> openssl req -new -newkey rsa:2048 -out cassy-.csr -keyout cassy-= .key -config cassy-.conf -subj "..." -nodes -sha256 >> >> The generated CSR was then signed by an internal root CA. Because we >> generated our files using openssl, we had to build our jks files by >> importing our certs into them. >> >> *Commands to generate truststore* >> >> We distribute this one file to all nodes. >> >> keytool -importcert >> -keystore generic-server-truststore.jks >> -alias rootCa >> -file rootCa.crt >> -noprompt >> -keypass omitted >> -storepass omitted >> >> *Commands to generate keystore* >> >> This was done one per region; but essentially we created a keystore with >> keytool, then deleted the key entry and then imported our key entry usin= g >> keytool from a pkcs12 file. >> >> keytool -genkeypair -keyalg RSA -alias cassy-${region} -keystore cassy-$= {region}.jks -storepass omitted -keypass omitted -validity 365 -keysize 204= 8 -dname "..." >> >> keytool -delete -alias cassy-${region} -keystore cassy-${region}.jks -st= orepass omitted >> >> openssl pkcs12 -export -in signed_certs/${region}.pem -inkey keys/cassan= dra.${region}.key -name cassy-${region} -out ${region}.p12 >> >> keytool -importkeystore -deststorepass omitted -destkeystore cassy-${reg= ion}.jks -srckeystore ${region}.p12 -srcstoretype PKCS12 >> >> keytool -importcert -keystore cassy-${region}.jks -alias rootCa -file ca= .crt -noprompt -keypass omitted -storepass omitted >> >> Looking back at this, I don't remember why we used keytool to generate a >> keypair/keystore, then deleted and imported. I think it was because the >> keytool importkeystore command refused to run if the keystore didn't >> already exist. >> >> *ca.crt and pem file* >> >> The ca.crt file contains the root certificate and the intermediate >> certificate that was used to sign the CSR. The pem file contains the sig= ned >> CSR returned to us, the intermediate cert, and the root CA (in that orde= r). >> >> *openssl verify ca.crt and pem* >> >> openssl verify -CAfile ca.crt us-west-2.pem >> signed_certs/us-west-2.pem: OK >> >> *Command output after enabling encryption* >> >> *nodetool status (output truncated)* >> >> Datacenter: us-east >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> Status=3DUp/Down >> |/ State=3DNormal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host ID = Rack >> ?N 52.44.11.221 ? 256 25.4% null = 1c >> ... >> ?N 52.204.232.195 ? 256 23.2% null = 1d >> Datacenter: us-west-2 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> Status=3DUp/Down >> |/ State=3DNormal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host ID = Rack >> ?N 34.209.2.144 ? 256 26.5% null = 2c >> UN 52.40.32.177 105.99 GiB 256 23.7% null = 2c >> ?N 34.210.109.203 ? 256 24.7% null = 2a >> ... >> >> With the online node being the node with encryption set. >> >> *cqlsh to localhost* >> >> cassy-node6:~$ cqlsh >> Connection error: ('Unable to connect to any servers', {'127.0.0.1': err= or(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection = refused")}) >> >> *cqlsh to remote node* Remote node is a node with encryption enabled >> >> cassy-node6:~$ cqlsh 10.0.2.7 >> Connection error: ('Unable to connect to any servers', {'10.0.2.7': erro= r(111, "Tried connecting to [('10.0.2.7', 9042)]. Last error: Connection re= fused")}) >> >> --0000000000008f207805910e3b24 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The version given by apt is 8u162-b12-1.=C2=A0 Which I thi= nk corresponds to openJDK-8-162.=C2=A0 When I run jrunscript -e 'print = (javax.crypto.Cipher.getMaxAllowedKeyLength("RC5") >=3D 256);&= #39; the command returns true.=C2=A0 Not sure if that is the best way to ve= rify JCE installed.

=
Michael Carlise

On Mo= n, Aug 26, 2019 at 5:47 PM Marc Selwan <marc.selwan@datastax.com> wrote:
which exact version of= OpenJDK are you using? Is it possible you don't have JCE on those node= s? (I believe more recent versions of Java 8 has this baked in so that migh= t not be it)


<= span style=3D"color:rgb(136,136,136)">Marc Selwan |=C2=A0DataStax | PM, Server Team=C2=A0<= b>|=C2=A0(925) 413-7079<= span style=3D"color:rgb(136,136,136)">=C2=A0|=C2=A0Twitter=C2=A0

=C2=A0 Quick links |=C2= =A0DataStax= =C2=A0|=C2=A0Training=C2=A0|=C2=A0Documentation=C2=A0|=C2=A0Downloads =C2=A0
=

=


On Mon, Aug 26, 2019 at 1:56 PM Michael Carli= se <mcarlise@salesforce.com.invalid> wrote:


However, I haven't gotten= any responses in over a week.=C2=A0 I'm going to post it here and mayb= e someone will have an idea on where I can look.

<= p style=3D"margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:i= nherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:inh= erit;font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font= -size:15px;vertical-align:baseline;box-sizing:inherit;clear:both;color:rgb(= 36,39,41)">We currently run a multi region cassandra cluster in AWS. It run= s in four regions, 12 nodes per region. It runs without node to node encryp= tion (or client encryption either). We are trying to enable inter datacente= r node to node encryption. However, when we flip encryption over we get an = exception that nodes are=C2=A0unable to gossip with any p= eers.

It could possibly be that we didn't build o= ur jks keystore/truststores correctly (more on how we built these files bel= ow). But, we additionally do not see intra datacenter communication working= (which should be set to unencrypted communication). Additionally, cqlsh ca= nnot connect to the node either; even though we have (by default)=C2=A0client_auth_required=C2=A0set to=C2=A0false.

ERROR [main] 2019-08-15 18:46:32,241 CassandraDaemon.java:749 - Exc=
eption encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:14=
35) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.checkForEndpointColl=
ision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.prepareToJoin(Storag=
eService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageSe=
rvice.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageSe=
rvice.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaem=
on.java:388) [apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraD=
aemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemo=
n.java:732) [apache-cassandra-3.11.4.jar:3.11.4]
INFO  [main] 2019-08-15 18:47:07,384 YamlConfigurationLoader.java:89 - Conf=
iguration location: file:/etc/cassandra/cassandra.yaml

Something to note is that this error message occur= s after a few minutes of the node being up. (i.e. there is a delay between = start up before this exception is thrown).

Information about our cassandra setup=

cassandra version: 3.11.4
J= DK version: openjdk-8.
Linux: Ubuntu 18.04 = (bionic).

cassandra.yaml

endpoint_snitch= : Ec2MultiRegionSnitch server_encryption_options: internode_encryption: dc keystore: <omitted> keystore_password: <omitted> truststore: <omitted> truststore_password: <omitted> client_encryption_options: enabled: false

cassandra-rackdc.properties

pr=
efer_local=3Dtrue

No o= bvious errors with SSH output

When starting cassand= ra with=C2=A0JVM_OPTS=3D"$JVM_OPTS -Djavax.net.debug= =3Dssl"=C2=A0added to=C2=A0cassandra-env.sh=C2=A0we see SSL logs printed to stdout (Note: Subject and Issuer were omitted on purpose).

found key for= : cassy-us-west-2 = = =20 adding as trusted cert: = = = =20 Subject: ... = = =20 Issuer: ... = = =20 Algorithm: RSA; Serial number: 0xdad28d843fc73325d4c1a75207d4e74 = = = =20 Valid from Fri May 27 00:00:00 UTC 2016 until Tue May 26 23:59:59 UTC 202= 6 =20 ... trigger seeding of SecureRandom done seeding SecureRandom =20

Looking at=C2=A0Java S= E SSL/TLS connection debugging, this looks correct. But to note, we see= this series of messages (along with the RSA key signature output) repeated= several times in rapid fire. We never observe any messages about the trust= store being added; however that might be something that occurs only on cli= ent initiation (?)

Additionally, we do see cassandra report = that the Encrypted Messaging service has been started.

INFO  [main] 2019-08-15 18:4=
5:31,022 MessagingService.java:704 - Starting Encrypted Messaging Service o=
n SSL port 7001

Does= n't appear to be a cassandra.yaml configuration problem

We can bring the node back online by simply configuring=C2=A0internode_encryption: none. This action seems to rule o= ut a broadcast_address or rpc_address configuration problem.

How we built our keystore/tr= uststores

We followed the basic template datastax d= ocs for=C2=A0preparing SSL certificates. One minor differen= ce was that our private key and CSRs were generated using=C2=A0openssl. One per each region (we plan to share key/signed cer= ts across nodes in regions). This was created using a command template as:<= /p>

openssl =
req -new -newkey rsa:2048 -out cassy-<region>.csr -keyout cassy-<r=
egion>.key -config cassy-<region>.conf -subj "..." -node=
s -sha256

The generated CSR was then signed by an internal r= oot CA. Because we generated our files using openssl, we had to build our j= ks files by importing our certs into them.

Com= mands to generate truststore

We distribute thi= s one file to all nodes.

keytool -importcert=20
    -keystore generic-server-truststore.jks=20
    -alias rootCa =20
    -file rootCa.crt=20
    -noprompt
    -keypass omitted=20
    -storepass omitted=20

Commands to generate keystore

This was done one per region; but essentially we cre= ated a keystore with keytool, then deleted the key entry and then imported = our key entry using keytool from a pkcs12 file.

keytool -genkeypair -keyalg RSA -al=
ias cassy-${region} -keystore cassy-${region}.jks -storepass omitted -keypa=
ss omitted -validity 365 -keysize 2048 -dname "..."=20

keytool -delete -alias cassy-${region} -keystore cassy-${region}.jks -store=
pass omitted

openssl pkcs12 -export -in signed_certs/${region}.pem -inkey keys/cassandra=
.${region}.key -name cassy-${region} -out ${region}.p12=20

keytool -importkeystore -deststorepass omitted -destkeystore cassy-${region=
}.jks -srckeystore ${region}.p12 -srcstoretype PKCS12=20

keytool -importcert -keystore cassy-${region}.jks -alias rootCa -file ca.cr=
t -noprompt -keypass omitted -storepass omitted=20

Looking back at this, I don't remember why we = used keytool to generate a keypair/keystore, then deleted and imported. I t= hink it was because the keytool importkeystore command refused to run if th= e keystore didn't already exist.

ca.crt an= d pem file

The=C2=A0ca.crt=C2=A0file contains the root certificate and the intermediate certific= ate that was used to sign the CSR. The pem file contains the signed CSR ret= urned to us, the intermediate cert, and the root CA (in that order).

openssl verify ca.crt and pem

openssl verify -=
CAfile ca.crt us-west-2.pem
signed_certs/us-west-2.pem: OK

Comm= and output after enabling encryption

nodetool status=C2=A0(output truncated)

Dat=
acenter: us-east                                                           =
                                    =20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D                  =
                   =20
Status=3DUp/Down                                          =20
|/ State=3DNormal/Leaving/Joining/Moving                  =20
--  Address         Load       Tokens       Owns (effective)  Host ID      =
                         Rack
?N  52.44.11.221    ?          256          25.4%             null         =
                         1c            =20
...
?N  52.204.232.195  ?          256          23.2%             null         =
                         1d            =20
Datacenter: us-west-2                                                      =
                                       =20
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Status=3DUp/Down                                          =20
|/ State=3DNormal/Leaving/Joining/Moving                  =20
--  Address         Load       Tokens       Owns (effective)  Host ID      =
                         Rack          =20
?N  34.209.2.144    ?          256          26.5%             null         =
                         2c            =20
UN  52.40.32.177    105.99 GiB  256          23.7%             null        =
                          2c           =20
?N  34.210.109.203  ?          256          24.7%             null         =
                         2a  =20
...                 =20

With the online node being the node with encryptio= n set.

cqlsh=C2=A0to = localhost

cassy-node6:~$ cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.=
0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)=
]. Last error: Connection refused")})

cqlsh=C2=A0= to remote node=C2=A0Remote node is a node with encryption ena= bled

cas=
sy-node6:~$ cqlsh 10.0.2.7
Connection error: ('Unable to connect to any servers', {'10.0.2=
.7': error(111, "Tried connecting to [('10.0.2.7', 9042)].=
 Last error: Connection refused")})
--0000000000008f207805910e3b24--