lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3892) Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
Date Thu, 09 Aug 2012 17:08:19 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431983#comment-13431983
] 

Michael McCandless commented on LUCENE-3892:
--------------------------------------------

OK indeed PFOR is slower for me too:

{noformat}
                Task    QPS base StdDev base    QPS pfor StdDev pfor      Pct diff
          HighPhrase        1.56        0.03        1.25        0.12  -28% -  -10%
           MedPhrase       13.05        0.10       10.50        0.58  -24% -  -14%
           LowPhrase       21.08        0.08       17.35        0.85  -22% -  -13%
          AndHighMed       73.78        0.66       62.50        1.68  -18% -  -12%
          AndHighLow      674.60        2.54      573.00       12.06  -17% -  -12%
         LowSpanNear        8.04        0.17        6.97        0.23  -17% -   -8%
         MedSpanNear        3.97        0.10        3.58        0.15  -15% -   -3%
     MedSloppyPhrase        7.58        0.11        6.93        0.14  -11% -   -5%
         AndHighHigh       25.71        0.47       23.58        0.61  -12% -   -4%
        HighSpanNear        1.42        0.04        1.31        0.05  -12% -   -1%
             MedTerm      155.44       18.75      144.46       12.33  -24% -   14%
            HighTerm       30.27        4.31       28.25        2.88  -26% -   19%
     LowSloppyPhrase        6.73        0.13        6.28        0.12  -10% -   -3%
          OrHighHigh        9.06        0.24        8.53        0.33  -11% -    0%
           OrHighLow       23.09        0.67       21.88        0.91  -11% -    1%
           OrHighMed       17.71        0.51       16.79        0.67  -11% -    1%
    HighSloppyPhrase        1.88        0.05        1.80        0.04   -9% -    0%
              IntNRQ        9.42        0.50        9.05        0.89  -17% -   11%
             Prefix3       72.67        2.42       70.42        3.61  -11% -    5%
              Fuzzy1       63.71        1.07       62.34        1.55   -6% -    1%
            Wildcard       45.25        0.99       44.28        1.55   -7% -    3%
            PKLookup      159.04        2.13      157.17        1.90   -3% -    1%
              Fuzzy2       62.51        2.28       63.40        1.65   -4% -    8%
             LowTerm      400.06       57.60      407.73       52.40  -22% -   34%
             Respell       56.72        3.19       59.83        2.10   -3% -   15%
{noformat}

I think we should replace Block with BlockPacked now?
                
> Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
> -------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3892
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3892
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>              Labels: gsoc2012, lucene-gsoc-12
>             Fix For: 4.1
>
>         Attachments: LUCENE-3892-BlockTermScorer.patch, LUCENE-3892-blockFor&hardcode(base).patch,
LUCENE-3892-blockFor&packedecoder(comp).patch, LUCENE-3892-blockFor-with-packedints-decoder.patch,
LUCENE-3892-blockFor-with-packedints-decoder.patch, LUCENE-3892-blockFor-with-packedints.patch,
LUCENE-3892-blockpfor.patch, LUCENE-3892-bulkVInt.patch, LUCENE-3892-direct-IntBuffer.patch,
LUCENE-3892-for&pfor-with-javadoc.patch, LUCENE-3892-handle_open_files.patch, LUCENE-3892-non-specialized.patch,
LUCENE-3892-pfor-compress-iterate-numbits.patch, LUCENE-3892-pfor-compress-slow-estimate.patch,
LUCENE-3892_for_byte[].patch, LUCENE-3892_for_int[].patch, LUCENE-3892_for_unfold_method.patch,
LUCENE-3892_pfor_unfold_method.patch, LUCENE-3892_pulsing_support.patch, LUCENE-3892_settings.patch,
LUCENE-3892_settings.patch
>
>
> On the flex branch we explored a number of possible intblock
> encodings, but for whatever reason never brought them to completion.
> There are still a number of issues opened with patches in different
> states.
> Initial results (based on prototype) were excellent (see
> http://blog.mikemccandless.com/2010/08/lucene-performance-with-pfordelta-codec.html
> ).
> I think this would make a good GSoC project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message