cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Voytek Jarnot <voytek.jar...@gmail.com>
Subject Re: SASI index creation assertion error
Date Sat, 05 Nov 2016 15:23:53 GMT
Indeed.  I did throw a comment on 11990 - not sure if that triggers emails
to those participants, but was hoping someone would take a look.

On Sat, Nov 5, 2016 at 2:26 AM, DuyHai Doan <doanduyhai@gmail.com> wrote:

> So from code review, the error message you get from the log is coming from
> the CASSANDRA-11990:  https://github.com/ifesdjeen/cassandra/commit/
> dc4ae57f452e19adbe5a6a2c85f8a4b5a24d4103#diff-
> eae81aa3b81f9b1e07b109c446447a50R357
>
> Now, it's just the consequence of the problem (throwing an assertion
> error), we have to dig further to understand why we fall into this situation
>
> On Sat, Nov 5, 2016 at 5:15 AM, Jonathan Haddad <jon@jonhaddad.com> wrote:
>
>> Can you file a Jira for this? Would be good to make sure 3.10 doesn't get
>> released with this bug.
>> On Fri, Nov 4, 2016 at 6:11 PM Voytek Jarnot <voytek.jarnot@gmail.com>
>> wrote:
>>
>>> Thought I'd follow-up to myself, in case anyone else comes across this
>>> problem.  I found a reasonably easy test case to reproduce the problem:
>>>
>>> This works in 3.9, but doesn't work in 3.10-snapshot:
>>>
>>> CREATE KEYSPACE vjtest WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '1'};
>>> use vjtest ;
>>> create table tester(id1 text, id2 text, id3 text, val1 text, primary
>>> key((id1, id2), id3));
>>> create custom index tester_idx_val1 on tester(val1) using '
>>> org.apache.cassandra.index.sasi.SASIIndex';
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','1-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','2-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','3-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','4-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','5-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','6-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','7-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','8-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','9-3','asdf');
>>>
>>> That's it - when Cassandra tries to flush all hell breaks loose (well,
>>> maaybe not, but an unhandled error gets logged).  Also, the index doesn't
>>> actually work subsequently.
>>>
>>> On Fri, Nov 4, 2016 at 3:58 PM, Voytek Jarnot <voytek.jarnot@gmail.com>
>>> wrote:
>>>
>>> Wondering if anyone has encountered the same...
>>>
>>> Full story and stacktraces below, short version is that creating a SASI
>>> index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
>>> the index on an empty table doesn't fail; however, soon after I start
>>> pumping data into the table similar problems occur.
>>>
>>> I created CASSANDRA-12877 for this, but am beginning to suspect it might
>>> be related to CASSANDRA-11990.  The thing that's throwing me is that I
>>> can't seem to duplicate this with a simple test table.
>>>
>>> Background:
>>>
>>> Ended up building/loading a 3.10-SNAPSHOT to try to get past
>>> CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
>>>
>>> 1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
>>> 2) created keyspace (SimpleStrategy, RF 1)
>>> 3) created table: (simplified version below, many more valX columns
>>> present)
>>>
>>> CREATE TABLE test_table (
>>>     id1 text,
>>>     id2 text,
>>>     id3 date,
>>>     id4 timestamp,
>>>     id5 text,
>>>     val1 text,
>>>     val2 text,
>>>     val3 text,
>>>     task_id text,
>>>     val4 text,
>>>     val5 text,
>>>     PRIMARY KEY ((id1, id2), id3, id4, id5)
>>> ) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
>>>
>>> 4) created materialized view:
>>>
>>> CREATE MATERIALIZED VIEW test_table_by_task_id AS
>>>     SELECT *
>>>     FROM test_table
>>>     WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND
>>> id4 IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
>>>     PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
>>>     WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5
>>> ASC)
>>>
>>> 5) inserted 27 million "rows" (i.e., unique values for id5)
>>> 6) create index attempt
>>>
>>> create custom index idx_test_table_val5 on test_table(val5) using '
>>> org.apache.cassandra.index.sasi.SASIIndex';
>>>
>>> 7) no error in cqlsh, but system.log shows many of the following:
>>>
>>> INFO  [SASI-General:1] 2016-11-04 13:46:47,578
>>> PerSSTableIndexWriter.java:277 - Flushed index segment
>>> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/
>>> mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-
>>> 149-big-SI_idx_test_table_val5.db_0, took 869 ms.
>>> ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229
>>> - Exception in thread Thread[SASI-General:1,5,main]
>>> java.lang.AssertionError: cannot have more than 8 overflow collisions
>>> per leaf, but had: 12
>>>     at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilde
>>> r$Leaf.createOverflowEntry(AbstractTokenTreeBuilder.java:357)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilde
>>> r$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder
>>> $DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilde
>>> r$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilde
>>> r.write(AbstractTokenTreeBuilder.java:90) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$Muta
>>> bleDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$Muta
>>> bleLevel.flush(OnDiskIndexBuilder.java:446)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$Muta
>>> bleLevel.add(OnDiskIndexBuilder.java:433) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.addT
>>> erm(OnDiskIndexBuilder.java:207) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.fini
>>> sh(OnDiskIndexBuilder.java:293) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.fini
>>> sh(OnDiskIndexBuilder.java:258) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.fini
>>> sh(OnDiskIndexBuilder.java:241) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$
>>> Index.lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> ~[na:1.8.0_101]
>>>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> ~[na:1.8.0_101]
>>>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_101]
>>>     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>>
>>> As well as some of these:
>>>
>>> ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142
>>> DataTracker.java:168 - Can't open index file at
>>> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/
>>> mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-
>>> 300-big-SI_idx_test_table_val5.db, skipping.
>>> java.lang.IllegalArgumentException: position: 3472329188772431788,
>>> limit: 8180
>>>     at org.apache.cassandra.index.sasi.utils.MappedBuffer.position(MappedBuffer.java:106)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.conf.DataTracker.getIndexes(DataTracker.java:150)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSST
>>> able(SASIIndexBuilder.java:156) [apache-cassandra-3.10-SNAPSHO
>>> T.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1583)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_101]
>>>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_101]
>>>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_101]
>>>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_101]
>>>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_101]
>>>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_101]
>>>     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>>
>>>
>>>
>

Mime
View raw message