incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ondřej Černoš <cern...@gmail.com>
Subject Re: java.io.IOException: FAILED_TO_UNCOMPRESS(5) exception when running nodetool rebuild
Date Wed, 27 Mar 2013 14:30:20 GMT
Hi Aaron,

I switched to 1.2.3 with no luck. I created
https://issues.apache.org/jira/browse/CASSANDRA-5391 describing the
problem. Maybe it's related to the EOFException problem, but I am not
sure - I don't know Cassandra internals well and I have never seen the
EOFException.

regards,

ondrej

On Tue, Mar 26, 2013 at 9:26 PM, aaron morton <aaron@thelastpickle.com> wrote:
> If you are still on 1.2.1 may be this
>
> https://issues.apache.org/jira/browse/CASSANDRA-5105
>
> Fixed in 1.2.2
>
> If you are on 1.2.3 there is also
> https://issues.apache.org/jira/browse/CASSANDRA-5381
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 26/03/2013, at 5:10 AM, Ondřej Černoš <cernoso@gmail.com> wrote:
>
> Hi all,
>
> I am still unable to move forward with this issue.
>
> - when I switch SSL off in inter-DC communication, nodetool rebuild  works
> well
> - when I switch internode_compression off, I still get
> java.io.IOException: FAILED_TO_UNCOMPRESS exception. Does
> internode_compression: none really switch off the snappy compression
> of the internode communication? The stacktrace - see the previous mail
> - clearly demonstrates some compression is involved
> - I managed to trigger another exception:
>
> java.lang.RuntimeException: javax.net.ssl.SSLException: bad record MAC
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1607)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:859)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 1 more
>
> I managed to trigger this exception only once however.
>
> The fact the transfer works when SSL is off and fails with SSL is
> another strange thing with this issue.
>
> Any ideas or hints?
>
> regards,
>
> Ondrej Cernos
>
> On Tue, Mar 19, 2013 at 5:51 PM, Ondřej Černoš <cernoso@gmail.com> wrote:
>
> Hi all,
>
> I am running into strange error when bootstrapping Cassandra cluster
> in multiple datacenter setup.
>
> The setup is as follows: 3 nodes in AWS east, 3 nodes somewhere on
> Rackspace/Openstack. I use my own snitch based on EC2MultiRegionSnitch
> (it just adds some ec2 avalability zone parsing capabilities). Nodes
> in the cluster connect to each other and all seems ok.
>
> When I start the Rackspace cluster first, populate it with data and
> then let the AWS cluster bootstrap from it, it works great. However
> the other way round it just breaks.
>
> The breakage demonstrates as follows:
>
> - nodetool rebuild us-east command hangs
> - cassandra's log contains the following:
>
> 2013-03-19 12:42:15.796+0100 [Thread-14] [DEBUG]
> IncomingTcpConnection.java(63)
> org.apache.cassandra.net.IncomingTcpConnection: Connection version 6
> from ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/xxx.xxx.xxx.xxx
> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
> StreamInSession.java(104)
> org.apache.cassandra.streaming.StreamInSession: Adding file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
> to Stream Request queue
> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
> StreamInSession.java(104)
> org.apache.cassandra.streaming.StreamInSession: Adding file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-1-Data.db
> to Stream Request queue
> 2013-03-19 12:42:15.806+0100 [Thread-14] [DEBUG]
> IncomingStreamReader.java(112)
> org.apache.cassandra.streaming.IncomingStreamReader: Receiving stream
> 2013-03-19 12:42:15.807+0100 [Thread-14] [DEBUG]
> IncomingStreamReader.java(113)
> org.apache.cassandra.streaming.IncomingStreamReader: Creating file for
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2-Data.db
> with 7808 estimat
> ed keys
> 2013-03-19 12:42:15.808+0100 [Thread-14] [DEBUG]
> ColumnFamilyStore.java(863) org.apache.cassandra.db.ColumnFamilyStore:
> component=key_space Checking for sstables overlapping []
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Data.db
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-CompressionInfo.db
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-TOC.txt
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Filter.db
> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Index.db
> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] SSTable.java(154)
> org.apache.cassandra.io.sstable.SSTable: Deleted
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2
> 2013-03-19 12:42:15.963+0100 [Thread-14] [INFO]
> StreamInSession.java(136)
> org.apache.cassandra.streaming.StreamInSession: Streaming of file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
> sections=127 progress=81048/2444
> 2013-03-19 12:42:16.059+0100 [Thread-13] [DEBUG]
> IncomingTcpConnection.java(79)
> org.apache.cassandra.net.IncomingTcpConnection: IOException reading
> from socket; closing
> java.io.IOException: FAILED_TO_UNCOMPRESS(5)
>        at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
>        at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
>        at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
>        at
> org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:93)
>        at
> org.apache.cassandra.streaming.compress.CompressedInputStream.decompress(CompressedInputStream.java:101)
>        at
> org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:79)
>        at
> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
>        at
> org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(BytesReadTracker.java:140)
>        at
> org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:361)
>        at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
>        at
> org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:160)
>        at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
>        at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
>        at
> org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
>        at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
> 2013-03-19 12:42:15.971+0100 [Thread-16] [ERROR]
> CassandraDaemon.java(133)
> org.apache.cassandra.service.CassandraDaemon: Exception in thread
> Thread[Thread-16,5,main]
> java.lang.RuntimeException: java.net.SocketException: Socket closed
>        at com.google.common.base.Throwables.propagate(Throwables.java:160)
>        at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
>        at java.lang.Thread.run(Thread.java:679)
> Caused by: java.net.SocketException: Socket closed
>        at java.net.SocketInputStream.socketRead0(Native Method)
>        at java.net.SocketInputStream.read(SocketInputStream.java:146)
>        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
>        at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
>        at sun.security.ssl.InputRecord.read(InputRecord.java:509)
>        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850)
>        at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807)
>        at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
>        at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
>        at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>        ... 1 more
>
> - the SSTable file is not downloaded.
>
> To make it even more confusing, when I tried just one node in AWS and
> one on Rackspace, the rebuild ran well. Also for two nodes. Only with
> three on AWS side the process failed.
>
> It's cassandra 1.2.1 running on
>
> java version "1.6.0_24"
> OpenJDK Runtime Environment (IcedTea6 1.11.8)
> (rhel-1.56.1.11.8.el6_3-x86_64)
> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
>
> SSL is configured for inter-DC communication.
>
> Any hints?
>
> regards,
>
> Ondrej Cernos
>
>

Mime
View raw message