cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
Date Mon, 16 May 2016 01:34:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284053#comment-15284053
] 

Stefania edited comment on CASSANDRA-11749 at 5/16/16 1:33 AM:
---------------------------------------------------------------

Thank you Norman.

You can use [this branch|https://github.com/apache/cassandra/compare/trunk...stef1927:11749-cqlsh-2.1].
It's configured to run Cassandra with SSL, it contains the test files and it links to Netty
4.0.36 (rather than 4.0.23). It's otherwise identical to cassandra-2.1 HEAD (the workaround
mentioned above has been commented out). 

Here are the instructions:

* Dependencies: Java 8 JDK, Python 2.7, ant 1.9+
* Get the branch: {{git clone http://github.com/stef1927/cassandra.git --branch 11749-cqlsh-2.1
--single-branch}}
* Set the {{CASSANDRA_DIR}} environment to the location of the branch and add {{CASSANDRA_DIR/bin}}
to the PATH
* Build: {{ant build}}
* If you need an IntelliJ project: {{ant generate-idea-files}} or Eclipse: {{ant generate-eclipse-files}}
* Generate the certificates by following [these instructions|http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html].
These are the certificates you should end up with:
{code}
keystore.node0
node0.cer
node0.cer.pem
node0.key.pem
node0.p12
truststore.node0
{code}
* Edit {{$CASSANDRA_DIR/cqlshrc}} and {{$CASSANDRA_DIR/conf/cassandra.yaml}} to point to your
certificates. The easiest is to search for my absolute path {{/home/stefi}} and change all
occurrences. There are 3 occurrences in {{cqlshrc}} and 2 in {{cassandra.yaml}}.
* Download the [JCE|http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html],
unzip and copy the 2 jars to {{$JAVA_HOME/jre/lib/security/}}
* Set any additional JVM properties via the {{JVM_OPTS}} environment variable, for example:
{{export JVM_OPTS=-Djavax.net.debug=ssl}}
* Launch cassandra in the foreground: {{cassandra -f}}. Stop with CTRL-C.
* If you need to run in IntelliJ you can use the Cassandra run config.
* The log file containing the exception is {{CASSANDRA_DIR/logs/system.log}}
* Run the test with {{cqlsh --debug --ssl --cqlshrc=./conf/cqlshrc -f kv.cql}}


This is a sample output when it works:

{code}
Using CQL driver: <module 'cassandra' from '/home/stefi/git/cstar/cassandra/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/__init__.py'>
Using connect timeout: 5 seconds
Reading options from the command line: {'header': 'true', 'numprocesses': '1'}
Using options: '{'header': 'true', 'numprocesses': '1'}'
Using 1 child processes

Starting copy of cvs_copy_ks.kv with columns ['key', 'value'].
Closing queues...; Rate:      12 rows/s; Avg. rate:      12 rows/s
Processed: 3 rows; Rate:       6 rows/s; Avg. rate:       8 rows/s
3 rows imported from 1 files in 0.358 seconds (0 skipped).

 key | value
-----+-------
   1 |   'a'
   2 |   'b'
   3 |   'c'

(3 rows)
{code}

This is a sample output when it fails, plus the exception will be visible in logs/system.log:

{code}
stefi@cuoricina:~/git/cstar/cassandra$ cqlsh --debug --ssl --cqlshrc=./conf/cqlshrc -f kv.cql
Using CQL driver: <module 'cassandra' from '/home/stefi/git/cstar/cassandra/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/__init__.py'>
Using connect timeout: 5 seconds
Reading options from the command line: {'header': 'true', 'numprocesses': '1'}
Using options: '{'header': 'true', 'numprocesses': '1'}'
Using 1 child processes

Starting copy of cvs_copy_ks.kv with columns ['key', 'value'].
Closing queues...; Rate:       9 rows/s; Avg. rate:       9 rows/s
Processed: 3 rows; Rate:       4 rows/s; Avg. rate:       7 rows/s
3 rows imported from 1 files in 0.449 seconds (0 skipped).
kv.cql:6:NoHostAvailable: ('Unable to complete the operation against any hosts', {})
kv.cql:7:NoHostAvailable: ('Unable to complete the operation against any hosts', {})
{code}

You should be able to reproduce this fairly easily since the workaround has been commented
out. I typically run it 5 or 6 times before reproducing it. To give you some context on the
test, {{copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true' and numprocesses=1;}}
will spawn a Python child process to import kv.csv into Cassandra. This command works but
the two following commands fail with {{NoHostAvailable}}, which indicate that the server did
not respond to cqlsh, plus we see the exception in the logs. You also find a file called {{loop.sh}}
if you want to run the test several times.

I hope I haven't forgotten any steps, if you run into trouble do not hesitate to let me know.
The instructions on generating certificates have a couple of typos, unfortunately I did not
save the exact commands I've used. You are probably familiar with those commands but if not
let me know and I'll recreate the certificates and give you the exact commands.


was (Author: stefania):
Thank you Norman.

You can use [this branch|https://github.com/apache/cassandra/compare/trunk...stef1927:11749-cqlsh-2.1].
It's configured to run Cassandra with SSL, it contains the test files and it links to Netty
4.0.36 (rather than 4.0.23). It's otherwise identical to cassandra-2.1 HEAD (the workaround
mentioned above has been commented out). 

Here are the instructions:

* Dependencies: Java 8 JDK, Python 2.7, ant 1.9+
* Get the branch: {{git clone http://github.com/stef1927/cassandra.git --branch 11749-cqlsh-2.1
--single-branch}}
* Set the {{CASSANDRA_DIR}} environment to the location of the branch and add {{CASSANDRA_DIR/bin}}
to the PATH
* Build: {{ant build}}
* If you need an IntelliJ project: {{ant generate-idea-files}} or Eclipse: {{ant generate-eclipse-files}}
* Generate the certificates by following [these instructions|http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html].
These are the certificates you should end up with:
{code}
keystore.node0
node0.cer
node0.cer.pem
node0.key.pem
node0.p12
truststore.node0
{code}
* Edit {{$CASSANDRA_DIR/cqlshrc}} and {{$CASSANDRA_DIR/conf/cassandra.yaml}} to point to your
certificates. The easiest is to search for my absolute path {{/home/stefi}} and change all
occurrences. There are 3 occurrences in {{cqlshrc}} and 2 in {{cassandra.yaml}}.
* Download the [JCE|http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html],
unzip and copy the 2 jars to {{$JAVA_HOME/jre/lib/security/}}
* Set any additional JVM properties via the {{JVM_OPTS}} environment variable, for example:
{{export JVM_OPTS=-Djavax.net.debug=ssl}}
* Launch cassandra in the foreground: {{cassandra -f}}. Stop with CTRL-C.
* If you need to run in IntelliJ you can you the Cassandra run config.
* The log file containing the exception is {{CASSANDRA_DIR/logs/system.log}}
* Run the test with {{cqlsh --debug --ssl --cqlshrc=./conf/cqlshrc -f kv.cql}}


This is a sample output when it works:

{code}
Using CQL driver: <module 'cassandra' from '/home/stefi/git/cstar/cassandra/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/__init__.py'>
Using connect timeout: 5 seconds
Reading options from the command line: {'header': 'true', 'numprocesses': '1'}
Using options: '{'header': 'true', 'numprocesses': '1'}'
Using 1 child processes

Starting copy of cvs_copy_ks.kv with columns ['key', 'value'].
Closing queues...; Rate:      12 rows/s; Avg. rate:      12 rows/s
Processed: 3 rows; Rate:       6 rows/s; Avg. rate:       8 rows/s
3 rows imported from 1 files in 0.358 seconds (0 skipped).

 key | value
-----+-------
   1 |   'a'
   2 |   'b'
   3 |   'c'

(3 rows)
{code}

This is a sample output when it fails, plus the exception will be visible in logs/system.log:

{code}
stefi@cuoricina:~/git/cstar/cassandra$ cqlsh --debug --ssl --cqlshrc=./conf/cqlshrc -f kv.cql
Using CQL driver: <module 'cassandra' from '/home/stefi/git/cstar/cassandra/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/__init__.py'>
Using connect timeout: 5 seconds
Reading options from the command line: {'header': 'true', 'numprocesses': '1'}
Using options: '{'header': 'true', 'numprocesses': '1'}'
Using 1 child processes

Starting copy of cvs_copy_ks.kv with columns ['key', 'value'].
Closing queues...; Rate:       9 rows/s; Avg. rate:       9 rows/s
Processed: 3 rows; Rate:       4 rows/s; Avg. rate:       7 rows/s
3 rows imported from 1 files in 0.449 seconds (0 skipped).
kv.cql:6:NoHostAvailable: ('Unable to complete the operation against any hosts', {})
kv.cql:7:NoHostAvailable: ('Unable to complete the operation against any hosts', {})
{code}

You should be able to reproduce this fairly easily since the workaround has been commented
out. I typically run it 5 or 6 times before reproducing it. To give you some context on the
test, {{copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true' and numprocesses=1;}}
will spawn a Python child process to import kv.csv into Cassandra. This command works but
the two following commands fail with {{NoHostAvailable}}, which indicate that the server did
not respond to cqlsh, plus we see the exception in the logs. You also find a file called {{loop.sh}}
if you want to run the test several times.

I hope I haven't forgotten any steps, if you run into trouble do not hesitate to let me know.
The instructions on generating certificates have a couple of typos, unfortunately I did not
save the exact commands I've used. You are probably familiar with those commands but if not
let me know and I'll recreate the certificates and give you the exact commands.

> CQLSH gets SSL exception following a COPY FROM
> ----------------------------------------------
>
>                 Key: CASSANDRA-11749
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>            Reporter: Stefania
>            Assignee: Stefania
>             Fix For: 2.1.x
>
>         Attachments: stdout.txt.zip, stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally results
in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 - Unexpected exception
during request; channel = [id: 0xeb75e05d, /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad record MAC
>         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
>         at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[na:1.8.0_91]
>         at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[na:1.8.0_91]
>         at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981) ~[na:1.8.0_91]
>         at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[na:1.8.0_91]
>         at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[na:1.8.0_91]
>         at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
>         at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
>         ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
>         at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219) ~[na:1.8.0_91]
>         at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177) ~[na:1.8.0_91]
>         at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974) ~[na:1.8.0_91]
>         ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql 
> create keyspace if not exists cvs_copy_ks with replication = {'class': 'SimpleStrategy',
'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuoricina:~/git/cstar/cassandra$ cat kv.c
> kv.cql  kv.csv  
> cat kv.csv 
> key,value
> 1,'a'
> 2,'b'
> 3,'c'
> {code}
> The COPY FROM succeeds, however the following select does not. 
> The easiest way to reproduce this is to restart the Cassandra process, it seems to happen
in preference after a restart.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message