We deleted and recreated those CFs before moving into
production mode. 
We have a wiener. 

The comparator is applying the current schema to the byte value read from disk (schema on read) which describes a value with more than 2 components. It's then trying to apply the current schema so it can type cast the bytes for comparison. 

Something must have gone wrong in the "deleted" part of your statement above. We do not store schema with data, so this a problem of changing the schema in an incompatible way with existing data. 

nodetool scrub is probably your best bet. I've not checked that it handles this specific problem, but in general it will drop rows from SSTables that cannot be read or have some other problem. Best thing to do is snapshot and copy the data from one prod node to a QA box and run some tests.

hope that helps. 

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton

On 3/04/2013, at 2:11 AM, Dane Miller <dane@optimalsocial.com> wrote:

On Mon, Apr 1, 2013 at 10:19 PM, aaron morton <aaron@thelastpickle.com> wrote:
ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-232,5,main]
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
      at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
      at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
      at
com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
      at
org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)

Something odd in the schema world perhaps.

Has the schema changed recently?

No, not recently.  But during development we experimented with other
comparator types for those CFs.  More info below.

Do yo have more than one schema in the cluster ? (describe cluster in
cassandra-cli)

I don't think so, that command in cassandra-cli shows just a single
schema in the cluster:

Cluster Information:
  Snitch: org.apache.cassandra.locator.Ec2Snitch
  Partitioner: org.apache.cassandra.dht.RandomPartitioner
  Schema versions:
126b31ad-3660-3831-9d4f-c6763c9acc97: [ ...ip list... ]


This error happened on a CF where the composite is: (Integer, UTF8)

I'm a bit stumped about how we could get the an index==2 in that code
pathway.  See here:
https://github.com/apache/cassandra/blob/cassandra-1.2.1/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java

...start at line 63 in compare.

My Java is terrible, but all of our CompositeTypes are composites of
only two types.  Thus the counter 'i' should never get up to 2 (it is
used to access by index to individual comparator types within the
composite), unless the value of the first two components in each of
the column names being compared are equal, which should be impossible.

During development we experimented with other comparator types for
those CFs.  We deleted and recreated those CFs before moving into
production mode.  Is there a chance Cassandra 'remembers' these old
types from a deleted CF that shares a name with an existing CF?  Could
that be causing improper parsing of comparator column names?

That we enter this code pathway from a section that seems to want to
clean up tombstones makes me think this is a possibility, that there
is a tombstone somewhere whose composite column name is causing
issues.

Dane