cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kurt Greaves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-14437) SSTableLoader does not work when "internode_encryption : all" is set
Date Tue, 08 May 2018 05:54:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466914#comment-16466914
] 

Kurt Greaves commented on CASSANDRA-14437:
------------------------------------------

This seems like you don't have internode encryption enabled on your nodes. You need to have
(more or less) the same internode_encryption settings for sstableloader and your nodes. {{none}}
shouldn't work with {{all}} or vice versa.

> SSTableLoader does not work when "internode_encryption : all" is set
> --------------------------------------------------------------------
>
>                 Key: CASSANDRA-14437
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14437
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>            Reporter: Paul Cheon
>            Priority: Major
>             Fix For: 3.11.2
>
>
> I am trying to use sstableloader to restore snapshot.
> If "internode_encryption :  all" is set, then it does not work and complain with below
error messages.  I initiated sstableloader from 10.1.10.203 (yvr-paul-cas003), so 10.1.10.203
worked fine, but the the other two nodes (10.1.10.201 & 10.1.10.202 failed)
> {noformat}
> pcheon@yvr-paul-cas003:~/t$ sstableloader -v -d 10.1.10.203 office_audit/log/ -f /etc/cassandra/cassandra.yaml
-u pcheon -pw `cat .secret`
> WARN  17:23:45,166 Small commitlog volume detected at /var/lib/cassandra/commitlog; setting
commitlog_total_space_in_mb to 2316.  You can override this in cassandra.yaml
> WARN  17:23:45,170 Small cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in_mb
to 1158.  You can override this in cassandra.yaml
> WARN  17:23:45,285 Only 5.318GiB free across all data volumes. Consider adding more capacity
to your cluster or removing obsolete snapshots
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of /home/pcheon/t/office_audit/log/mc-1083-big-Data.db /home/pcheon/t/office_audit/log/mc-1100-big-Data.db
/home/pcheon/t/office_audit/log/mc-1101-big-Data.db /home/pcheon/t/office_audit/log/mc-257-big-Data.db
/home/pcheon/t/office_audit/log/mc-984-big-Data.db  to [/10.1.10.201, /10.1.10.203, /10.1.10.202]
> ERROR 17:23:49,460 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming error occurred
on session with peer 10.1.10.201
> java.net.SocketException: Invalid argument or cannot assign requested address
> 	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_112]
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[na:1.8.0_112]
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_112]
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
> 	at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
> 	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_112]
> 	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) ~[na:1.8.0_112]
> 	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169)
~[na:1.8.0_112]
> 	at org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) ~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56)
~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282)
~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) ~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263)
[apache-cassandra-3.11.2.jar:3.11.2]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.2.jar:3.11.2]
> 	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> ERROR 17:23:49,458 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming error occurred
on session with peer 10.1.10.202
> java.net.SocketException: Invalid argument or cannot assign requested address
> 	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_112]
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[na:1.8.0_112]
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_112]
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
> 	at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
> 	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_112]
> 	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) ~[na:1.8.0_112]
> 	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169)
~[na:1.8.0_112]
> 	at org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) ~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56)
~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282)
~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) ~[apache-cassandra-3.11.2.jar:3.11.2]
> 	at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263)
[apache-cassandra-3.11.2.jar:3.11.2]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
> 	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.2.jar:3.11.2]
> 	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
> progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
> progress: [/10.1.10.203]0:0/5 7  % total: 7% 13.858MiB/s (avg: 2.147MiB/s)
> progress: [/10.1.10.203]0:0/5 15 % total: 15% 41.587MiB/s (avg: 4.083MiB/s)
> progress: [/10.1.10.203]0:0/5 23 % total: 23% 75.631MiB/s (avg: 5.964MiB/s)
> progress: [/10.1.10.203]0:0/5 31 % total: 31% 34.837MiB/s (avg: 7.523MiB/s)
> progress: [/10.1.10.203]0:0/5 39 % total: 39% 21.321MiB/s (avg: 8.641MiB/s)
> progress: [/10.1.10.203]0:1/5 41 % total: 41% 17.665MiB/s (avg: 8.766MiB/s)
> progress: [/10.1.10.203]0:1/5 49 % total: 49% 17.841MiB/s (avg: 9.557MiB/s)
> progress: [/10.1.10.203]0:1/5 56 % total: 56% 75.654MiB/s (avg: 10.889MiB/s)
> progress: [/10.1.10.203]0:1/5 64 % total: 64% 21.252MiB/s (avg: 11.583MiB/s)
> progress: [/10.1.10.203]0:1/5 72 % total: 72% 33.267MiB/s (avg: 12.472MiB/s)
> progress: [/10.1.10.203]0:1/5 80 % total: 80% 22.969MiB/s (avg: 13.060MiB/s)
> progress: [/10.1.10.203]0:2/5 81 % total: 81% 6.191MiB/s (avg: 12.951MiB/s)
> progress: [/10.1.10.203]0:3/5 81 % total: 81% 100.225MiB/s (avg: 12.994MiB/s)
> progress: [/10.1.10.203]0:3/5 89 % total: 89% 12.939MiB/s (avg: 12.989MiB/s)
> progress: [/10.1.10.203]0:3/5 97 % total: 97% 65.803MiB/s (avg: 13.899MiB/s)
> progress: [/10.1.10.203]0:4/5 99 % total: 99% 69.864MiB/s (avg: 14.038MiB/s)
> progress: [/10.1.10.203]0:5/5 100% total: 100% 56.930MiB/s (avg: 14.141MiB/s)
> progress: [/10.1.10.203]0:5/5 100% total: 100% 0.000KiB/s (avg: 13.373MiB/s)
> WARN  17:23:54,923 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Stream failed
> Streaming to the following hosts failed:
> [/10.1.10.201, /10.1.10.202]
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> 	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> 	at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
> 	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> 	at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> 	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> 	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> 	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> 	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> 	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> 	at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
> 	at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
> 	at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
> 	at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
> 	at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
> 	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
> 	at java.lang.Thread.run(Thread.java:745)
> Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
> 	at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:114)
> 	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
> Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> 	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> 	at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
> 	... 1 more
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> 	at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> 	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> 	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> 	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> 	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> 	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> 	at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
> 	at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
> 	at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
> 	at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
> 	at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
> 	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> However, if I set the "internode_encryption : none", then it works fine as expected
> I do not know why it does not work.  I guess somehow sstableloader does not know how
to talk with other Cassandra through encrypted channel?
> Would you help?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message