ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-232,5,main]
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
       at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
       at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
       at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
       at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
Something odd in the schema world perhaps. 

Has the schema changed recently? 
Do yo have more than one schema in the cluster ? (describe cluster in cassandra-cli)

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton

On 2/04/2013, at 4:34 AM, Dane Miller <dane@optimalsocial.com> wrote:

I hit some more errors while running repair.  Repair starts streaming,
and then hangs.  On 2 of the 3 nodes involved in the repair, "nodetool
netstats" show no streams.  On the 3rd node there is an error in the
logs and "nodetool netstats" show several streams that appear hung --
no progress in about 1 hour.  The error message from the log is below,
also viewable here: http://pastebin.com/JTcXgkNR.  Any ideas how to
fix this?

ERROR [Thread-233] 2013-04-01 22:22:21,759 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-233,5,main]
java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException
       at com.google.common.base.Throwables.propagate(Throwables.java:160)
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
       at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.channels.AsynchronousCloseException
       at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
       at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:270)
       at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
       at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
       at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
       at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
       ... 1 more
ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-232,5,main]
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
       at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
       at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
       at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
       at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
       at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
       at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
       at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:128)
       at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:125)
       at java.util.TreeMap.compare(TreeMap.java:1173)
       at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1243)
       at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1677)
       at java.util.TreeMap.tailMap(TreeMap.java:890)
       at java.util.TreeSet.tailSet(TreeSet.java:333)
       at java.util.TreeSet.tailSet(TreeSet.java:366)
       at org.apache.cassandra.db.RangeTombstone$Tracker.update(RangeTombstone.java:206)
       at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:180)
       at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:276)
       at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
       at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
       at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
       at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
       at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)



Details:
cassandra  1.2.2 - single token per node
Ubuntu 10.10 x86_64
EC2 m1.large
Cassandra max heap: 4G


Dane