lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Drob (JIRA)" <>
Subject [jira] [Commented] (LUCENE-7440) Document skipping on large indexes is broken
Date Mon, 12 Sep 2016 16:59:21 GMT


Mike Drob commented on LUCENE-7440:

bq. Regarding the 1.8B docs number... at least in my tests I saw the top-level skip distance
of ~268M w/ the default codec. Subtracting this from MAX_INT gives around 1.8B, which is around
the number I saw prior to the overflow. To hit the bug, one also needs to be doing large skips
toward the end of the index as well, in order to use the top level(s) of the multi-level skip
list. Having a conjunction query of a highly unique term (or clause) in conjunction with a
common term has a good chance of triggering (example: +timestamp:39520928456494 +doctype:common)
Would this be faster to test if we configure a larger top-level skip distance? i.e. set up
a skip distance of ~1B and then we'd only need to get to ~1.1B docs indexed (40% fewer docs,
theoretically 40% faster?) or even set up a skip distance of ~2B to only need to index very
few documents?

Maybe this idea should be split into a separate issue to focus on improving the test?

> Document skipping on large indexes is broken
> --------------------------------------------
>                 Key: LUCENE-7440
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/search
>    Affects Versions: 2.2
>            Reporter: Yonik Seeley
>            Assignee: Yonik Seeley
>            Priority: Critical
>             Fix For: master (7.0), 6.3, 6.2.1
>         Attachments: LUCENE-7440.patch, LUCENE-7440.patch
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted queries,
join queries, etc) can trigger this bug on a sufficiently large index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present since inception
(Lucene 2.2).  It may not manifest until one has a single segment with more than ~1.8B documents,
and a large skip is performed on that segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> 	at org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(
> 	at
> 	at org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(
> 	at org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(
> 	at org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(
> 	at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(
> 	at YCS_skip7$1.testSkip(
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter: null:java.lang.ArrayIndexOutOfBoundsException:
>  at org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(
>  at
>  at org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(
>  at org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(
>  at org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(
>  at org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(
>  at
> [...]
>  at
> [...]
>  at org.apache.solr.core.SolrCore.execute(
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message