incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathijs Vogelzang <>
Subject Streaming fails with inter node encryption on
Date Mon, 25 Mar 2013 09:45:54 GMT
Hi all,

We have a 6-node cassandra cluster which has worked fine for a long
time through upgrades starting from 0.8.x to 1.1.x. Recently we
upgraded to 1.2.2, and since then streaming repair doesn't work
anymore (everything else works, gossip, serving Thrift queries etc.).
We upgraded to 1.2.3, upgraded the JDK to the latest version (1.7u17),
but nothing helped. The only error message in the logs is the
following pasted below:

 INFO [AntiEntropyStage:1] 2013-03-25 09:30:33,493 (line 162) Streaming to /
 INFO [Streaming to /] 2013-03-25 09:30:33,859 (line 50) Need to re-stream file
/var/lib/cassandra/data/....db to /
 INFO [Streaming to /] 2013-03-25 09:30:33,994 (line 50) Need to re-stream file
/var/lib/cassandra/data/....db to /
 INFO [Streaming to /] 2013-03-25 09:30:34,190 (line 50) Need to re-stream file
/var/lib/cassandra/data/.....db to /
ERROR [Streaming to /] 2013-03-25 09:30:34,474 (line 164) Exception in thread Thread[Streaming
to /,5,main]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$
Caused by:
        at org.apache.cassandra.streaming.FileStreamTask.receiveReply(
        at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(
        ... 3 more

Subsequently the repair command hangs, and the nodes start running out
of memory after a few cycles with the heap being full of Merkle tree
related datastructures.

We've now discovered that when we turn internode encryption off then
the streaming works again. Is there something that could explain why
the regular internode network traffic works (else thrift queries
should also fail), but the streaming doesn't?

Our internode encryption settings were:
    internode_encryption: all
    keystore: conf/.keystore
    keystore_password: xxxxxxxx
    truststore: conf/.truststore
    truststore_password: xxxxxxxx
    protocol: TLS
    algorithm: SunX509
    store_type: JKS
    cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA]

Best regards,


View raw message