lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Han Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LUCENE-3892) Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
Date Wed, 08 Aug 2012 03:37:11 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397228#comment-13397228
] 

Han Jiang edited comment on LUCENE-3892 at 8/8/12 3:35 AM:
-----------------------------------------------------------

And result for PFor(blocksize=128):
{noformat}
                Task    QPS Base StdDev Base    QPS PFor StdDev PFor      Pct diff
              Phrase        4.87        0.36        3.39        0.18  -38% -  -20% (-47% -
 -25%)
          AndHighMed       27.78        2.35       21.13        0.52  -31% -  -14% (-37% -
 -15%)
            SpanNear        2.70        0.14        2.20        0.11  -26% -   -9% (-36% -
 -13%)
        SloppyPhrase        4.17        0.15        3.77        0.21  -17% -    0% (-30% -
  -6%)
             Respell       39.97        1.56       37.65        1.95  -14% -    3% (-15% -
   2%)
            Wildcard       19.08        0.77       18.33        0.92  -12% -    5% (-17% -
   3%)
              Fuzzy1       42.29        1.13       40.78        1.44   -9% -    2% (-11% -
   1%)
         AndHighHigh        7.61        0.55        7.45        0.08   -9% -    6% (-19% -
   6%)
              Fuzzy2       15.79        0.55       15.64        0.70   -8% -    7% (-11% -
   6%)
            PKLookup       86.71        2.13       88.92        2.24   -2% -    7% ( -2% -
   7%)
         TermGroup1M       13.04        0.23       14.03        0.40    2% -   12% (  1% -
   9%)
              IntNRQ        3.97        0.48        4.35        0.61  -15% -   41% (-16% -
  24%)
      TermBGroup1M1P       21.04        0.35       23.20        0.60    5% -   14% (  0% -
  14%)
        TermBGroup1M       19.27        0.47       21.28        0.84    3% -   17% (  1% -
  10%)
          OrHighHigh        4.13        0.47        4.63        0.27   -5% -   34% (-14% -
  27%)
           OrHighMed        4.95        0.59        5.58        0.34   -5% -   35% (-14% -
  27%)
             Prefix3       30.33        1.36       34.26        2.14    1% -   25% ( -6% -
  20%)
                Term       41.99        1.19       50.75        1.72   13% -   28% (  2% -
  26%)
{noformat}
-It works, and it is quite interesting that StdDev for Term query is reduced significantly.
- (same as last comment, when comparing two versions directly(method call vs. unfolded, the
improvement is somewhat noisy))
                
      was (Author: billy):
    And result for PFor(blocksize=128):
{noformat}
                Task    QPS Base StdDev Base    QPS PFor StdDev PFor      Pct diff
              Phrase        4.87        0.36        3.39        0.18  -38% -  -20% (-47% -
 -25%)
          AndHighMed       27.78        2.35       21.13        0.52  -31% -  -14% (-37% -
 -15%)
            SpanNear        2.70        0.14        2.20        0.11  -26% -   -9% (-36% -
 -13%)
        SloppyPhrase        4.17        0.15        3.77        0.21  -17% -    0% (-30% -
  -6%)
             Respell       39.97        1.56       37.65        1.95  -14% -    3% (-15% -
   2%)
            Wildcard       19.08        0.77       18.33        0.92  -12% -    5% (-17% -
   3%)
              Fuzzy1       42.29        1.13       40.78        1.44   -9% -    2% (-11% -
   1%)
         AndHighHigh        7.61        0.55        7.45        0.08   -9% -    6% (-19% -
   6%)
              Fuzzy2       15.79        0.55       15.64        0.70   -8% -    7% (-11% -
   6%)
            PKLookup       86.71        2.13       88.92        2.24   -2% -    7% ( -2% -
   7%)
         TermGroup1M       13.04        0.23       14.03        0.40    2% -   12% (  1% -
   9%)
              IntNRQ        3.97        0.48        4.35        0.61  -15% -   41% (-16% -
  24%)
      TermBGroup1M1P       21.04        0.35       23.20        0.60    5% -   14% (  0% -
  14%)
        TermBGroup1M       19.27        0.47       21.28        0.84    3% -   17% (  1% -
  10%)
          OrHighHigh        4.13        0.47        4.63        0.27   -5% -   34% (-14% -
  27%)
           OrHighMed        4.95        0.59        5.58        0.34   -5% -   35% (-14% -
  27%)
             Prefix3       30.33        1.36       34.26        2.14    1% -   25% ( -6% -
  20%)
                Term       41.99        1.19       50.75        1.72   13% -   28% (  2% -
  26%)
{noformat}
It works, and it is quite interesting that StdDev for Term query is reduced significantly.
 
                  
> Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
> -------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3892
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3892
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>              Labels: gsoc2012, lucene-gsoc-12
>             Fix For: 4.1
>
>         Attachments: LUCENE-3892-BlockTermScorer.patch, LUCENE-3892-blockFor&hardcode(base).patch,
LUCENE-3892-blockFor&packedecoder(comp).patch, LUCENE-3892-blockFor-with-packedints-decoder.patch,
LUCENE-3892-blockFor-with-packedints-decoder.patch, LUCENE-3892-blockFor-with-packedints.patch,
LUCENE-3892-direct-IntBuffer.patch, LUCENE-3892-for&pfor-with-javadoc.patch, LUCENE-3892-handle_open_files.patch,
LUCENE-3892-pfor-compress-iterate-numbits.patch, LUCENE-3892-pfor-compress-slow-estimate.patch,
LUCENE-3892_for_byte[].patch, LUCENE-3892_for_int[].patch, LUCENE-3892_for_unfold_method.patch,
LUCENE-3892_pfor_unfold_method.patch, LUCENE-3892_pulsing_support.patch, LUCENE-3892_settings.patch,
LUCENE-3892_settings.patch
>
>
> On the flex branch we explored a number of possible intblock
> encodings, but for whatever reason never brought them to completion.
> There are still a number of issues opened with patches in different
> states.
> Initial results (based on prototype) were excellent (see
> http://blog.mikemccandless.com/2010/08/lucene-performance-with-pfordelta-codec.html
> ).
> I think this would make a good GSoC project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message