Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of cernoso@gmail.com designates
 209.85.223.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAgSJSX5oPK1EA6gOayr29B3kLm14UMstOTSesgu-L7SQboMQA@mail.gmail.com>
References: 
 <CAAgSJSX5oPK1EA6gOayr29B3kLm14UMstOTSesgu-L7SQboMQA@mail.gmail.com>
Date: Mon, 25 Mar 2013 17:10:11 +0100
Message-ID: 
 <CAAgSJSUwE2iqqouwqr9YN5SF5wNnHGT+=NfX43fU1fc86P_t3g@mail.gmail.com>
Subject: Re: java.io.IOException: FAILED_TO_UNCOMPRESS(5) exception when
 running nodetool rebuild
From: =?UTF-8?B?T25kxZllaiDEjGVybm/FoQ==?= <cernoso@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi all,

I am still unable to move forward with this issue.

- when I switch SSL off in inter-DC communication, nodetool rebuild  works =
well
- when I switch internode_compression off, I still get
java.io.IOException: FAILED_TO_UNCOMPRESS exception. Does
internode_compression: none really switch off the snappy compression
of the internode communication? The stacktrace - see the previous mail
- clearly demonstrates some compression is involved
- I managed to trigger another exception:

java.lang.RuntimeException: javax.net.ssl.SSLException: bad record MAC
	at com.google.common.base.Throwables.propagate(Throwables.java:160)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
	at java.lang.Thread.run(Thread.java:662)
Caused by: javax.net.ssl.SSLException: bad record MAC
	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:164=
9)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:160=
7)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.jav=
a:859)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl=
.java:755)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75=
)
	at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.ru=
nMayThrow(CompressedInputStream.java:151)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	... 1 more

I managed to trigger this exception only once however.

The fact the transfer works when SSL is off and fails with SSL is
another strange thing with this issue.

Any ideas or hints?

regards,

Ondrej Cernos

On Tue, Mar 19, 2013 at 5:51 PM, Ond=C5=99ej =C4=8Cerno=C5=A1 <cernoso@gmai=
l.com> wrote:
> Hi all,
>
> I am running into strange error when bootstrapping Cassandra cluster
> in multiple datacenter setup.
>
> The setup is as follows: 3 nodes in AWS east, 3 nodes somewhere on
> Rackspace/Openstack. I use my own snitch based on EC2MultiRegionSnitch
> (it just adds some ec2 avalability zone parsing capabilities). Nodes
> in the cluster connect to each other and all seems ok.
>
> When I start the Rackspace cluster first, populate it with data and
> then let the AWS cluster bootstrap from it, it works great. However
> the other way round it just breaks.
>
> The breakage demonstrates as follows:
>
> - nodetool rebuild us-east command hangs
> - cassandra's log contains the following:
>
> 2013-03-19 12:42:15.796+0100 [Thread-14] [DEBUG]
> IncomingTcpConnection.java(63)
> org.apache.cassandra.net.IncomingTcpConnection: Connection version 6
> from ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/xxx.xxx.xxx.xxx
> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
> StreamInSession.java(104)
> org.apache.cassandra.streaming.StreamInSession: Adding file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-i=
b-2-Data.db
> to Stream Request queue
> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
> StreamInSession.java(104)
> org.apache.cassandra.streaming.StreamInSession: Adding file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-i=
b-1-Data.db
> to Stream Request queue
> 2013-03-19 12:42:15.806+0100 [Thread-14] [DEBUG]
> IncomingStreamReader.java(112)
> org.apache.cassandra.streaming.IncomingStreamReader: Receiving stream
> 2013-03-19 12:42:15.807+0100 [Thread-14] [DEBUG]
> IncomingStreamReader.java(113)
> org.apache.cassandra.streaming.IncomingStreamReader: Creating file for
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-t=
mp-ib-2-Data.db
> with 7808 estimat
> ed keys
> 2013-03-19 12:42:15.808+0100 [Thread-14] [DEBUG]
> ColumnFamilyStore.java(863) org.apache.cassandra.db.ColumnFamilyStore:
> component=3Dkey_space Checking for sstables overlapping []
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Data.db
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-CompressionInfo.db
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-TOC.txt
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Filter.db
> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Index.db
> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] SSTable.java(154)
> org.apache.cassandra.io.sstable.SSTable: Deleted
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-t=
mp-ib-2
> 2013-03-19 12:42:15.963+0100 [Thread-14] [INFO]
> StreamInSession.java(136)
> org.apache.cassandra.streaming.StreamInSession: Streaming of file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-i=
b-2-Data.db
> sections=3D127 progress=3D81048/2444
> 2013-03-19 12:42:16.059+0100 [Thread-13] [DEBUG]
> IncomingTcpConnection.java(79)
> org.apache.cassandra.net.IncomingTcpConnection: IOException reading
> from socket; closing
> java.io.IOException: FAILED_TO_UNCOMPRESS(5)
>         at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:7=
8)
>         at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
>         at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
>         at org.apache.cassandra.io.compress.SnappyCompressor.uncompress(S=
nappyCompressor.java:93)
>         at org.apache.cassandra.streaming.compress.CompressedInputStream.=
decompress(CompressedInputStream.java:101)
>         at org.apache.cassandra.streaming.compress.CompressedInputStream.=
read(CompressedInputStream.java:79)
>         at java.io.DataInputStream.readUnsignedShort(DataInputStream.java=
:337)
>         at org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(=
BytesReadTracker.java:140)
>         at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(Byte=
BufferUtil.java:361)
>         at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(=
ByteBufferUtil.java:371)
>         at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(I=
ncomingStreamReader.java:160)
>         at org.apache.cassandra.streaming.IncomingStreamReader.read(Incom=
ingStreamReader.java:122)
>         at org.apache.cassandra.net.IncomingTcpConnection.stream(Incoming=
TcpConnection.java:226)
>         at org.apache.cassandra.net.IncomingTcpConnection.handleStream(In=
comingTcpConnection.java:166)
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcp=
Connection.java:66)
> 2013-03-19 12:42:15.971+0100 [Thread-16] [ERROR]
> CassandraDaemon.java(133)
> org.apache.cassandra.service.CassandraDaemon: Exception in thread
> Thread[Thread-16,5,main]
> java.lang.RuntimeException: java.net.SocketException: Socket closed
>         at com.google.common.base.Throwables.propagate(Throwables.java:16=
0)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable=
.java:32)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: java.net.SocketException: Socket closed
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:146)
>         at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
>         at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554=
)
>         at sun.security.ssl.InputRecord.read(InputRecord.java:509)
>         at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:8=
50)
>         at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.ja=
va:807)
>         at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
>         at org.apache.cassandra.streaming.compress.CompressedInputStream$=
Reader.runMayThrow(CompressedInputStream.java:151)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable=
.java:28)
>         ... 1 more
>
> - the SSTable file is not downloaded.
>
> To make it even more confusing, when I tried just one node in AWS and
> one on Rackspace, the rebuild ran well. Also for two nodes. Only with
> three on AWS side the process failed.
>
> It's cassandra 1.2.1 running on
>
> java version "1.6.0_24"
> OpenJDK Runtime Environment (IcedTea6 1.11.8) (rhel-1.56.1.11.8.el6_3-x86=
_64)
> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
>
> SSL is configured for inter-DC communication.
>
> Any hints?
>
> regards,
>
> Ondrej Cernos