lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3892) Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
Date Thu, 09 Aug 2012 15:44:19 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431914#comment-13431914
] 

Michael McCandless commented on LUCENE-3892:
--------------------------------------------

I compared Block w/ BlockPacked, but set acceptableOverheadRatio to 0 for a fairer test:

{noformat}
                Task    QPS base StdDev base    QPS pack StdDev pack      Pct diff
    HighSloppyPhrase        1.94        0.01        1.91        0.05   -4% -    2%
           LowPhrase       21.05        0.07       20.84        0.37   -3% -    1%
           MedPhrase       13.05        0.04       12.93        0.23   -3% -    1%
            Wildcard       43.87        2.76       43.49        2.10  -11% -   10%
              IntNRQ        8.88        1.39        8.83        0.78  -21% -   28%
              Fuzzy1       63.07        1.96       62.78        1.46   -5% -    5%
     LowSloppyPhrase        6.92        0.01        6.91        0.13   -2% -    1%
             Prefix3       71.38        5.20       71.35        3.17  -10% -   12%
            PKLookup      157.00        1.78      158.01        2.01   -1% -    3%
          AndHighLow      668.76        4.82      674.80        7.48    0% -    2%
          HighPhrase        1.56        0.03        1.58        0.03   -3% -    5%
     MedSloppyPhrase        7.71        0.03        7.80        0.11    0% -    2%
          AndHighMed       74.05        0.49       75.35        0.36    0% -    2%
         AndHighHigh       25.92        0.30       26.78        0.19    1% -    5%
             Respell       57.07        2.70       59.20        1.80   -3% -   12%
              Fuzzy2       60.81        2.92       63.32        1.68   -3% -   12%
          OrHighHigh        8.99        0.17        9.39        0.11    1% -    7%
           OrHighMed       17.65        0.37       18.52        0.13    2% -    7%
         MedSpanNear        3.90        0.17        4.11        0.09   -1% -   12%
           OrHighLow       22.99        0.51       24.22        0.15    2% -    8%
        HighSpanNear        1.40        0.06        1.48        0.03    0% -   12%
         LowSpanNear        7.84        0.31        8.32        0.17    0% -   12%
             LowTerm      406.02       28.53      444.21       37.75   -6% -   27%
             MedTerm      149.83        8.11      167.60       15.06   -3% -   28%
            HighTerm       29.57        1.67       33.42        3.20   -3% -   31%
{noformat}

Curiously it seems even faster than w/ acceptableOverheadRatio=0.2!  But it makes it clear
we should do a hard cutover.
                
> Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
> -------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3892
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3892
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>              Labels: gsoc2012, lucene-gsoc-12
>             Fix For: 4.1
>
>         Attachments: LUCENE-3892-BlockTermScorer.patch, LUCENE-3892-blockFor&hardcode(base).patch,
LUCENE-3892-blockFor&packedecoder(comp).patch, LUCENE-3892-blockFor-with-packedints-decoder.patch,
LUCENE-3892-blockFor-with-packedints-decoder.patch, LUCENE-3892-blockFor-with-packedints.patch,
LUCENE-3892-blockpfor.patch, LUCENE-3892-bulkVInt.patch, LUCENE-3892-direct-IntBuffer.patch,
LUCENE-3892-for&pfor-with-javadoc.patch, LUCENE-3892-handle_open_files.patch, LUCENE-3892-non-specialized.patch,
LUCENE-3892-pfor-compress-iterate-numbits.patch, LUCENE-3892-pfor-compress-slow-estimate.patch,
LUCENE-3892_for_byte[].patch, LUCENE-3892_for_int[].patch, LUCENE-3892_for_unfold_method.patch,
LUCENE-3892_pfor_unfold_method.patch, LUCENE-3892_pulsing_support.patch, LUCENE-3892_settings.patch,
LUCENE-3892_settings.patch
>
>
> On the flex branch we explored a number of possible intblock
> encodings, but for whatever reason never brought them to completion.
> There are still a number of issues opened with patches in different
> states.
> Initial results (based on prototype) were excellent (see
> http://blog.mikemccandless.com/2010/08/lucene-performance-with-pfordelta-codec.html
> ).
> I think this would make a good GSoC project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message