lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3892) Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
Date Mon, 30 Jul 2012 23:38:34 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425388#comment-13425388
] 

Michael McCandless commented on LUCENE-3892:
--------------------------------------------

I tested Block vs BlockPacked as checked in.

On a Westmere Xeon machine (Java 1.7.0_04):

{noformat}
                Task    QPS base StdDev base     QPS for  StdDev for      Pct diff
          AndHighMed       15.14        0.14       13.78        0.13  -10% -   -7%
        SloppyPhrase        2.55        0.11        2.33        0.09  -15% -   -1%
          OrHighHigh        3.75        0.16        3.44        0.09  -14% -   -1%
            Wildcard        8.44        0.01        7.78        0.28  -11% -   -4%
            SpanNear        1.11        0.04        1.03        0.04  -13% -    0%
             Prefix3       17.91        0.08       16.63        0.50  -10% -   -3%
           OrHighMed       11.35        0.65       10.63        0.44  -15% -    3%
              IntNRQ        6.73        0.03        6.32        0.27  -10% -   -1%
        TermBGroup1M        3.87        0.03        3.68        0.04   -6% -   -3%
         AndHighHigh        4.86        0.09        4.63        0.03   -7% -   -2%
              Phrase        1.10        0.06        1.05        0.06  -14% -    6%
                Term        7.86        0.03        7.52        0.04   -5% -   -3%
      TermBGroup1M1P        4.65        0.12        4.49        0.06   -6% -    0%
         TermGroup1M        2.97        0.04        2.88        0.02   -4% -   -1%
              Fuzzy1       71.22        1.93       71.02        1.44   -4% -    4%
              Fuzzy2       49.76        1.33       49.90        1.23   -4% -    5%
             Respell       76.23        2.67       76.93        2.67   -5% -    8%
            PKLookup      161.89        3.28      168.28        7.87   -2% -   11%
{noformat}

And on an desktop Ivy Bridge (Java 1.7.0_04):
{noformat}
                Task    QPS base StdDev base     QPS for  StdDev for      Pct diff
          AndHighMed       17.32        0.12       15.41        0.03  -11% -  -10%
        SloppyPhrase        2.74        0.21        2.56        0.11  -16% -    5%
              Phrase        1.32        0.07        1.23        0.06  -15% -    3%
            Wildcard        9.65        0.11        9.08        0.12   -8% -   -3%
            SpanNear        1.20        0.01        1.13        0.01   -7% -   -3%
         AndHighHigh        5.32        0.03        5.04        0.02   -6% -   -4%
             Prefix3       18.93        0.20       18.04        0.24   -6% -   -2%
              IntNRQ        7.79        0.13        7.48        0.13   -7% -    0%
                Term        9.48        0.10        9.15        0.43   -8% -    2%
        TermBGroup1M        4.74        0.05        4.59        0.12   -6% -    0%
           OrHighMed       13.01        0.24       12.60        0.55   -9% -    2%
          OrHighHigh        4.08        0.05        3.97        0.17   -8% -    2%
         TermGroup1M        3.30        0.03        3.22        0.07   -5% -    0%
      TermBGroup1M1P        5.52        0.11        5.42        0.22   -7% -    4%
            PKLookup      194.62        4.43      193.44        5.07   -5% -    4%
              Fuzzy1       79.23        1.31       79.21        0.96   -2% -    2%
             Respell       78.97        1.04       79.87        1.15   -1% -    3%
              Fuzzy2       56.17        0.93       56.82        0.64   -1% -    4%
{noformat}

So packed is still behind ...
                
> Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
> -------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3892
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3892
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>              Labels: gsoc2012, lucene-gsoc-12
>             Fix For: 4.1
>
>         Attachments: LUCENE-3892-BlockTermScorer.patch, LUCENE-3892-blockFor&hardcode(base).patch,
LUCENE-3892-blockFor&packedecoder(comp).patch, LUCENE-3892-blockFor-with-packedints-decoder.patch,
LUCENE-3892-blockFor-with-packedints-decoder.patch, LUCENE-3892-blockFor-with-packedints.patch,
LUCENE-3892-direct-IntBuffer.patch, LUCENE-3892-for&pfor-with-javadoc.patch, LUCENE-3892-handle_open_files.patch,
LUCENE-3892-pfor-compress-iterate-numbits.patch, LUCENE-3892-pfor-compress-slow-estimate.patch,
LUCENE-3892_for_byte[].patch, LUCENE-3892_for_int[].patch, LUCENE-3892_for_unfold_method.patch,
LUCENE-3892_pfor_unfold_method.patch, LUCENE-3892_pulsing_support.patch, LUCENE-3892_settings.patch,
LUCENE-3892_settings.patch
>
>
> On the flex branch we explored a number of possible intblock
> encodings, but for whatever reason never brought them to completion.
> There are still a number of issues opened with patches in different
> states.
> Initial results (based on prototype) were excellent (see
> http://blog.mikemccandless.com/2010/08/lucene-performance-with-pfordelta-codec.html
> ).
> I think this would make a good GSoC project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message