incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dane Miller <d...@optimalsocial.com>
Subject IndexOutOfBoundsException during repair, streaming
Date Mon, 01 Apr 2013 23:04:33 GMT
I hit some more errors while running repair.  Repair starts streaming,
and then hangs.  On 2 of the 3 nodes involved in the repair, "nodetool
netstats" show no streams.  On the 3rd node there is an error in the
logs and "nodetool netstats" show several streams that appear hung --
no progress in about 1 hour.  The error message from the log is below,
also viewable here: http://pastebin.com/JTcXgkNR.  Any ideas how to
fix this?

ERROR [Thread-233] 2013-04-01 22:22:21,759 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-233,5,main]
java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.channels.AsynchronousCloseException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:270)
        at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
        at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        ... 1 more
ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-232,5,main]
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
        at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
        at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:128)
        at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:125)
        at java.util.TreeMap.compare(TreeMap.java:1173)
        at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1243)
        at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1677)
        at java.util.TreeMap.tailMap(TreeMap.java:890)
        at java.util.TreeSet.tailSet(TreeSet.java:333)
        at java.util.TreeSet.tailSet(TreeSet.java:366)
        at org.apache.cassandra.db.RangeTombstone$Tracker.update(RangeTombstone.java:206)
        at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:180)
        at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:276)
        at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
        at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
        at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)



Details:
cassandra  1.2.2 - single token per node
Ubuntu 10.10 x86_64
EC2 m1.large
Cassandra max heap: 4G


Dane

Mime
View raw message