incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathijs Vogelzang <math...@apptornado.com>
Subject Streaming fails with inter node encryption on
Date Mon, 25 Mar 2013 09:45:54 GMT
Hi all,

We have a 6-node cassandra cluster which has worked fine for a long
time through upgrades starting from 0.8.x to 1.1.x. Recently we
upgraded to 1.2.2, and since then streaming repair doesn't work
anymore (everything else works, gossip, serving Thrift queries etc.).
We upgraded to 1.2.3, upgraded the JDK to the latest version (1.7u17),
but nothing helped. The only error message in the logs is the
following pasted below:

 INFO [AntiEntropyStage:1] 2013-03-25 09:30:33,493
StreamOutSession.java (line 162) Streaming to /xxx.xxx.xxx.xxx
 INFO [Streaming to /10.181.129.193:1] 2013-03-25 09:30:33,859
StreamReplyVerbHandler.java (line 50) Need to re-stream file
/var/lib/cassandra/data/....db to /xxx.xxx.xxx.xxx
 INFO [Streaming to /10.181.129.193:1] 2013-03-25 09:30:33,994
StreamReplyVerbHandler.java (line 50) Need to re-stream file
/var/lib/cassandra/data/....db to /xxx.xxx.xxx.xxx
 INFO [Streaming to /10.181.129.193:1] 2013-03-25 09:30:34,190
StreamReplyVerbHandler.java (line 50) Need to re-stream file
/var/lib/cassandra/data/.....db to /xxx.xxx.xxx.xxx
ERROR [Streaming to /10.181.129.193:1] 2013-03-25 09:30:34,474
CassandraDaemon.java (line 164) Exception in thread Thread[Streaming
to /xxx.xxx.xxx.xxx:1,5,main]
java.lang.RuntimeException: java.io.EOFException
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193)
        at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114)
        at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        ... 3 more

Subsequently the repair command hangs, and the nodes start running out
of memory after a few cycles with the heap being full of Merkle tree
related datastructures.

We've now discovered that when we turn internode encryption off then
the streaming works again. Is there something that could explain why
the regular internode network traffic works (else thrift queries
should also fail), but the streaming doesn't?

Our internode encryption settings were:
server_encryption_options:
    internode_encryption: all
    keystore: conf/.keystore
    keystore_password: xxxxxxxx
    truststore: conf/.truststore
    truststore_password: xxxxxxxx
    protocol: TLS
    algorithm: SunX509
    store_type: JKS
    cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA]


Best regards,

Mathijs

Mime
View raw message