lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Han Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4283) Support more frequent skip with Block Postings Format
Date Fri, 03 Aug 2012 19:25:02 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428311#comment-13428311
] 

Han Jiang commented on LUCENE-4283:
-----------------------------------

bq. Can't we call skipWriter.bufferSkip every skipInterval docs (and pass it lastDocID, etc.)?
Then it can write the skip point immediately.
Hmm, actually, no. We can't predict the df when buffering skip data, therefore, we may save
extra skip data for the vInt block. For example, df=128+33 and interval=32.

bq. Also, in BlockPostingsReader, why do we need a separate docBufferOffset? Can't we just
set docBufferUpto to wherever (36, 64, 96) we had skipped to within the block?
Yes, you're right! I'll clean up those codes.
                
> Support more frequent skip with Block Postings Format
> -----------------------------------------------------
>
>                 Key: LUCENE-4283
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4283
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Han Jiang
>            Priority: Minor
>         Attachments: LUCENE-4283-buggy.patch, LUCENE-4283-buggy.patch
>
>
> This change works on the new bulk branch.
> Currently, our BlockPostingsFormat only supports skipInterval==blockSize. Every time
the skipper reaches the last level 0 skip point, we'll have to decode a whole block to read
doc/freq data. Also,  a higher level skip list will be created only for those df>blockSize^k,
which means for most terms, skipping will just be a linear scan. If we increase current blockSize
for better bulk i/o performance, current skip setting will be a bottleneck. 
> For ForPF, the encoded block can be easily splitted if we set skipInterval=32*k. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message