cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Semeria (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-5905) Cassandra crashes on restart with "IndexOutOfBoundsException: index (2) must be less than size (2)"
Date Tue, 20 Aug 2013 13:24:52 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744942#comment-13744942
] 

David Semeria edited comment on CASSANDRA-5905 at 8/20/13 1:24 PM:
-------------------------------------------------------------------

It's worth mentioning that only one of the nodes initially crashed - and the crash happened
much later, when the system was idle.
The real problems arose when I attempted to restart the crashed node.
Given the above, it's hard to know exactly what caused the cluster to become unstable.
However, I have a feeling it could be linked to the creation of secondary indexes.
I was doing multiple runs of the script below (using the native Java driver) which dropped
a table and then re-created it along with 3 secondary indexes.
It's worth noting that the script only dropped the table, not the indexes.
Nevertheless, no exception was thrown when the 3 indexes were recreated (I assumed dropping
the table automatically dropped the indexes, is this correct?)

DROP TABLE demo.test1;

CREATE TABLE demo.test1 (uuid text,ctg text,tgt text,uid text,local_ctg text,local_tgt text,local_uid
text,tag text,rating int,timestamp bigint, 
PRIMARY KEY (ctg, uuid) 
);

CREATE INDEX idx_local_uid ON demo.test1 (local_uid);
CREATE INDEX idx_local_tgt ON demo.test1 (local_tgt);
CREATE INDEX idx_tag ON demo.test1 (tag);

INSERT INTO demo.test1
(ctg,uuid,local_uid,local_tgt,tag,rating,timestamp)
VALUES ('cities','e6ceff8c-87f0-471f-a000-2650fb3b4291','david','nyc','id-0',1,1377003017428);

INSERT INTO demo.test1
(ctg,uuid,local_uid,local_tgt,tag,rating,timestamp)
VALUES ('cities','4179c256-4737-490e-ad67-499988333537','david','nyc','tag1',1,1377003017429);

INSERT INTO demo.test1
(ctg,uuid,local_uid,local_tgt,tag,rating,timestamp)
VALUES ('cities','8bcc2d96-8c3d-4aa5-9d90-1da470c8e95b','david','nyc','tg2',1,1377003017429);

[SNIP]

SELECT local_uid,local_tgt,tag,rating
FROM test1
WHERE ctg = 'cities' AND tag = 'tag1'
ALLOW FILTERING;

[OUTPUT AS EXPECTED]
                
      was (Author: hymanroth):
    It's worth mentioning that only one of the nodes crashed (and the crash happened when
the system was idle).
The problems arose when I attempted to restart the crashed node.
Given the above, it's hard to know exactly what caused the cluster to become unstable.
However, I have a feeling it could be linked to the creation of secondary indexes.
I was running the script below (using the native Java driver) which dropped a table and then
re-created it along with 3 secondary indexes.
It's worth noting that the script only dropped the table and not the indexes.
Nevertheless, no exception was thrown when the 3 indexes were recreated (I assumed dropping
the table automatically dropped the indexes, is this correct?)

DROP TABLE demo.test1;

CREATE TABLE demo.test1 (uuid text,ctg text,tgt text,uid text,local_ctg text,local_tgt text,local_uid
text,tag text,rating int,timestamp bigint, 
PRIMARY KEY (ctg, uuid) 
);

CREATE INDEX idx_local_uid ON demo.test1 (local_uid);
CREATE INDEX idx_local_tgt ON demo.test1 (local_tgt);
CREATE INDEX idx_tag ON demo.test1 (tag);

INSERT INTO demo.test1
(ctg,uuid,local_uid,local_tgt,tag,rating,timestamp)
VALUES ('cities','e6ceff8c-87f0-471f-a000-2650fb3b4291','david','nyc','id-0',1,1377003017428);

INSERT INTO demo.test1
(ctg,uuid,local_uid,local_tgt,tag,rating,timestamp)
VALUES ('cities','4179c256-4737-490e-ad67-499988333537','david','nyc','tag1',1,1377003017429);

INSERT INTO demo.test1
(ctg,uuid,local_uid,local_tgt,tag,rating,timestamp)
VALUES ('cities','8bcc2d96-8c3d-4aa5-9d90-1da470c8e95b','david','nyc','tg2',1,1377003017429);

[SNIP]

SELECT local_uid,local_tgt,tag,rating
FROM test1
WHERE ctg = 'cities' AND tag = 'tag1'
ALLOW FILTERING;

{0}  -m->  [local_uid]  --->  david
           [local_tgt]  --->  lon
           [tag      ]  --->  tag1
           [rating   ]  -i->  1

{1}  -m->  [local_uid]  --->  david
           [local_tgt]  --->  paris
           [tag      ]  --->  tag1
           [rating   ]  -i->  1

{2}  -m->  [local_uid]  --->  david
           [local_tgt]  --->  nyc
           [tag      ]  --->  tag1
           [rating   ]  -i->  1

{3}  -m->  [local_uid]  --->  paco
           [local_tgt]  --->  nyc
           [tag      ]  --->  tag1
           [rating   ]  -i->  1

{4}  -m->  [local_uid]  --->  frank
           [local_tgt]  --->  paris
           [tag      ]  --->  tag1
           [rating   ]  -i->  1

{5}  -m->  [local_uid]  --->  frank
           [local_tgt]  --->  lon
           [tag      ]  --->  tag1
           [rating   ]  -i->  1

{6}  -m->  [local_uid]  --->  frank
           [local_tgt]  --->  nyc
           [tag      ]  --->  tag1
           [rating   ]  -i->  1

{7}  -m->  [local_uid]  --->  paco
           [local_tgt]  --->  lon
           [tag      ]  --->  tag1
           [rating   ]  -i->  1

{8}  -m->  [local_uid]  --->  paco
           [local_tgt]  --->  paris
           [tag      ]  --->  tag1
           [rating   ]  -i->  1


                  
> Cassandra crashes on restart with "IndexOutOfBoundsException: index (2) must be less
than size (2)"
> ---------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5905
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5905
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 12.04 x64
> 3 nodes, 1GB each
>            Reporter: David Semeria
>             Fix For: 1.2.7
>
>
> All 3 nodes crash on restart with same error:
> INFO 13:31:05,272 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1376754824649.log
>  INFO 13:31:05,272 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1376754824650.log
>  INFO 13:31:08,267 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1376754824650.log
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> 	at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> 	at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> 	at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
> 	at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
> 	at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
> 	at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
> 	at java.util.TreeMap.compare(TreeMap.java:1188)
> 	at java.util.TreeMap.put(TreeMap.java:531)
> 	at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
> 	at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
> 	at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
> 	at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
> 	at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:376)
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:203)
> 	at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:98)
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
> 	at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:281)
> 	at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:358)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:601)
> 	at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
> Cannot load daemon
> Service exit with a return value of 3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message