Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Wed, 12 Jun 2013 08:43:20 +0000 (UTC)
From: "Stefan Pohl (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <JIRA.12652247.1370977233807.102863.1371026600040@arcas>
In-Reply-To: <JIRA.12652247.1370977233807@arcas>
References: <JIRA.12652247.1370977233807@arcas>
Subject: [jira] [Commented] (LUCENE-5052) bitset codec for off heap filters
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/LUCENE-5052?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D136=
81049#comment-13681049 ]=20

Stefan Pohl commented on LUCENE-5052:
-------------------------------------

The following paper might be informative in regard to this ticket (you can =
even go beyond maxdocs/8, if compared against VInt-coding):

A. Moffat and J. S. Culpepper. Hybrid Bitvector Index Compression. In Proce=
edings of the 12th Australasian Document Computing Symposium (ADCS 2007), D=
ecember 2007. pp 25-37.
available from: http://goanna.cs.rmit.edu.au/~e76763/publications.html

More generally, it would be nice to determine the PostingsListFormat depend=
ing on statistics of individual terms, not only per-field.
               =20
> bitset codec for off heap filters
> ---------------------------------
>
>                 Key: LUCENE-5052
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5052
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: core/codecs
>            Reporter: Mikhail Khludnev
>              Labels: features
>             Fix For: 5.0
>
>
> Colleagues,
> When we filter we don=E2=80=99t care any of scoring factors i.e. norms, p=
ositions, tf, but it should be fast. The obvious way to handle this is to d=
ecode postings list and cache it in heap (CachingWrappingFilter, Solr=E2=80=
=99s DocSet). Both of consuming a heap and decoding as well are expensive.=
=20
> Let=E2=80=99s write a posting list as a bitset, if df is greater than seg=
ment's maxdocs/8  (what about skiplists? and overall performance?).=20
> Beside of the codec implementation, the trickiest part to me is to design=
 API for this. How we can let the app know that a term query don=E2=80=99t =
need to be cached in heap, but can be held as an mmaped bitset?
> WDYT? =20

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org