Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Mon, 12 Sep 2016 19:05:21 +0000 (UTC)
From: "Yonik Seeley (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <JIRA.13003736.1473389550000.551160.1473707121666@Atlassian.JIRA>
In-Reply-To: <JIRA.13003736.1473389550000@Atlassian.JIRA>
References: <JIRA.13003736.1473389550000@Atlassian.JIRA> <JIRA.13003736.1473389550502@arcas>
Subject: [jira] [Commented] (LUCENE-7440) Document skipping on large indexes
 is broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 12 Sep 2016 19:05:24 -0000


    [ https://issues.apache.org/jira/browse/LUCENE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15484994#comment-15484994 ] 

Yonik Seeley commented on LUCENE-7440:
--------------------------------------

bq. Would this be faster to test if we configure a larger top-level skip distance?

The top-level skip distance sort of falls out from other factors, rather than being explicitly configured.
For a quicker, more thorough test, It would probably be good to somehow test the skip list logic itself w/o having it backed by an actual index.  Even with that, I think it's a good idea to also test real indexes.

> Document skipping on large indexes is broken
> --------------------------------------------
>
>                 Key: LUCENE-7440
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7440
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/search
>    Affects Versions: 2.2
>            Reporter: Yonik Seeley
>            Assignee: Yonik Seeley
>            Priority: Critical
>             Fix For: master (7.0), 6.3, 6.2.1
>
>         Attachments: LUCENE-7440.patch, LUCENE-7440.patch
>
>
> Large skips on large indexes fail.
> Anything that uses skips (such as a boolean query, filtered queries, faceted queries, join queries, etc) can trigger this bug on a sufficiently large index.
> The bug is a numeric overflow in MultiLevelSkipList that has been present since inception (Lucene 2.2).  It may not manifest until one has a single segment with more than ~1.8B documents, and a large skip is performed on that segment.
> Typical stack trace on Lucene7-dev:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 110
> 	at org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:297)
> 	at org.apache.lucene.store.DataInput.readVInt(DataInput.java:125)
> 	at org.apache.lucene.codecs.lucene50.Lucene50SkipReader.readSkipData(Lucene50SkipReader.java:180)
> 	at org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:163)
> 	at org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:133)
> 	at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:421)
> 	at YCS_skip7$1.testSkip(YCS_skip7.java:307)
> {code}
> Typical stack trace on Lucene4.10.3:
> {code}
> 6-08-31 18:57:17,460 ERROR org.apache.solr.servlet.SolrDispatchFilter: null:java.lang.ArrayIndexOutOfBoundsException: 75
>  at org.apache.lucene.codecs.MultiLevelSkipListReader$SkipBuffer.readByte(MultiLevelSkipListReader.java:301)
>  at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
>  at org.apache.lucene.codecs.lucene41.Lucene41SkipReader.readSkipData(Lucene41SkipReader.java:194)
>  at org.apache.lucene.codecs.MultiLevelSkipListReader.loadNextSkip(MultiLevelSkipListReader.java:168)
>  at org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo(MultiLevelSkipListReader.java:138)
>  at org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.advance(Lucene41PostingsReader.java:506)
>  at org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
> [...]
>  at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
> [...]
>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2004)
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org