cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Petrov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12877) SASI index throwing AssertionError on creation/flush
Date Tue, 08 Nov 2016 17:34:58 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648205#comment-15648205
] 

Alex Petrov commented on CASSANDRA-12877:
-----------------------------------------

I'm reopening it as after giving it more thought I realised that the problem with overflows
is much deeper than I initially thought. In the previous version, the collision would happen
only when there were multiple offsets per token (which would happen on murmur hash collision),
as there's a long set that holds partition position, that makes sure we're tracking unique
partition positions per sstable, and create overflow entries when there's more than one partition
position per token.

Introduction of the row offset now results into the problem that, when there are many equal
items in the same partition, we're reaching the overflow. Unfortunately, there's no quick
fix for that, although we already were discussing one of the changes that will solve this
problem.

> SASI index throwing AssertionError on creation/flush
> ----------------------------------------------------
>
>                 Key: CASSANDRA-12877
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12877
>             Project: Cassandra
>          Issue Type: Bug
>          Components: sasi
>         Environment: 3.9 and 3.10 tested on both linux and osx
>            Reporter: Voytek Jarnot
>            Assignee: Alex Petrov
>
> Possibly a 3.10 regression?  The exact test shown below does not error in 3.9.
> I built and installed a 3.10 snapshot (built 04-Nov-2016) to get around CASSANDRA-11670,
CASSANDRA-12689, and CASSANDRA-12223 which are holding me back when using 3.9.
> Now I'm able to make nodetool flush (or a scheduled flush) produce an unhandled error
easily with a SASI:
> {code}
> CREATE KEYSPACE vjtest WITH replication = {'class': 'SimpleStrategy', 'replication_factor':
'1'};
> use vjtest ;
> create table tester(id1 text, id2 text, id3 text, val1 text, primary key((id1, id2),
id3));
> create custom index tester_idx_val1 on tester(val1) using 'org.apache.cassandra.index.sasi.SASIIndex';
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','1-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','2-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','3-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','4-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','5-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','6-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','7-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','8-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','9-3','asdf');
> {code}
> Not enough going on here to trigger a flush, so following a manual {{nodetool flush vjtest}}
I get the following in {{system.log}}:
> {code}
> INFO  [MemtableFlushWriter:3] 2016-11-04 22:19:35,412 PerSSTableIndexWriter.java:284
- Scheduling index flush to /mydir/apache-cassandra-3.10-SNAPSHOT/data/data/vjtest/tester-6f1fdff0a30611e692c087673c5ef8d4/mc-1-big-SI_tester_idx_val1.db
> INFO  [SASI-Memtable:1] 2016-11-04 22:19:35,447 PerSSTableIndexWriter.java:335 - Index
flush to /mydir/apache-cassandra-3.10-SNAPSHOT/data/data/vjtest/tester-6f1fdff0a30611e692c087673c5ef8d4/mc-1-big-SI_tester_idx_val1.db
took 16 ms.
> ERROR [SASI-Memtable:1] 2016-11-04 22:19:35,449 CassandraDaemon.java:229 - Exception
in thread Thread[SASI-Memtable:1,5,RMI Runtime]
> java.lang.AssertionError: cannot have more than 8 overflow collisions per leaf, but had:
9
>     at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.createOverflowEntry(AbstractTokenTreeBuilder.java:357)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder$DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder.write(AbstractTokenTreeBuilder.java:90)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableLevel.flush(OnDiskIndexBuilder.java:446)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableLevel.finalFlush(OnDiskIndexBuilder.java:451)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:296)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:258)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:241)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$complete$1(PerSSTableIndexWriter.java:296)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_101]
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_101]
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_101]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
>     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> ERROR [MemtableFlushWriter:3] 2016-11-04 22:19:35,453 DataTracker.java:168 - Can't open
index file at /mydir/apache-cassandra-3.10-SNAPSHOT/data/data/vjtest/tester-6f1fdff0a30611e692c087673c5ef8d4/mc-1-big-SI_tester_idx_val1.db,
skipping.
> java.lang.IllegalArgumentException: position: 8314882962218811392, limit: 4113
>     at org.apache.cassandra.index.sasi.utils.MappedBuffer.position(MappedBuffer.java:106)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.conf.DataTracker.getIndexes(DataTracker.java:150)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69) [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.index.sasi.SASIIndex.handleNotification(SASIIndex.java:320)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.db.lifecycle.Tracker.notifyAdded(Tracker.java:421) [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.db.lifecycle.Tracker.replaceFlushed(Tracker.java:356) [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.db.compaction.CompactionStrategyManager.replaceFlushed(CompactionStrategyManager.java:317)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1574)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1197)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1100)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
>     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> ERROR [Reference-Reaper:1] 2016-11-04 22:21:59,610 Ref.java:224 - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@1fb888b0) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@293565751:/mydir/apache-cassandra-3.10-SNAPSHOT/data/data/vjtest/tester-6f1fdff0a30611e692c087673c5ef8d4/mc-1-big
was not released before the reference was garbage collected
> {code}
> Subsequently:
> {code}select * from tester where val1='asdf';{code} produces 0 rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message