lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Han Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LUCENE-3892) Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
Date Fri, 13 Jul 2012 16:01:35 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413781#comment-13413781
] 

Han Jiang edited comment on LUCENE-3892 at 7/13/12 4:00 PM:
------------------------------------------------------------

This patch cut the extra header and merge numBytes into header. Also, it store only one int
when a whole block share the same value. So no matter which strategy we use (d-gap, or d-gap
minus 1), it will work well.

And here are the changes between different methods, postingsformat=PFor:
{noformat}
method          base     all_0s  all_1s  all_Vs  all_Vs+header_cut
index time(s):  324     315     279     279     291
index size(MB): 591     591     589     589     577
{noformat}

postingsFormat=For: 
{noformat}
method          base     all_Vs+header_cut
index time(s):  250     251
index size(MB): 611     598
{noformat}

where raw refers to the version when numBits==0 isn't supported, all_0s refers to last patch,
all_Vs+header_cut refers to this patch. As for PFor, now the index size is almost equal to
Lucene40(590.7M vs 590.0M).
                
      was (Author: billy):
    This patch cut the extra header and merge numBytes into header. Also, it store only one
int when a whole block share the same value. So no matter which strategy we use (d-gap, or
d-gap minus 1), it will work well.

And here are the changes between different methods:
{noformat}
method          raw     all_0s  all_1s  all_Vs  all_Vs+header_cut
index time(s):  324     315     279     279     291
index size(MB): 591     591     589     589     577
{noformat}
where raw refers to the version when numBits==0 isn't supported, all_0s refers to last patch,
all_Vs+header_cut refers to this patch
                  
> Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
> -------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3892
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3892
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>              Labels: gsoc2012, lucene-gsoc-12
>             Fix For: 4.1
>
>         Attachments: LUCENE-3892-BlockTermScorer.patch, LUCENE-3892-direct-IntBuffer.patch,
LUCENE-3892-for&pfor-with-javadoc.patch, LUCENE-3892-for&pfor-with-javadoc.patch,
LUCENE-3892-for&pfor-with-javadoc.patch, LUCENE-3892-for&pfor.patch, LUCENE-3892-handle_open_files.patch,
LUCENE-3892_for.patch, LUCENE-3892_for_byte[].patch, LUCENE-3892_for_int[].patch, LUCENE-3892_for_unfold_method.patch,
LUCENE-3892_pfor.patch, LUCENE-3892_pfor.patch, LUCENE-3892_pfor.patch, LUCENE-3892_pfor_unfold_method.patch,
LUCENE-3892_pulsing_support.patch, LUCENE-3892_settings.patch, LUCENE-3892_settings.patch
>
>
> On the flex branch we explored a number of possible intblock
> encodings, but for whatever reason never brought them to completion.
> There are still a number of issues opened with patches in different
> states.
> Initial results (based on prototype) were excellent (see
> http://blog.mikemccandless.com/2010/08/lucene-performance-with-pfordelta-codec.html
> ).
> I think this would make a good GSoC project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message