lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3932) Improve load time of .tii files
Date Thu, 29 Mar 2012 18:40:22 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241497#comment-13241497
] 

Michael McCandless commented on LUCENE-3932:
--------------------------------------------

Nice.  I'd love to know how trunk handles all these terms (we have a more memory efficient
terms dict/index in 4.0).

bq. After the change the big time waste is converting the terms from utf8 to utf16 when reading
from the .tii file, and then back to utf8 when writing to the in memory store.

What %tg of the time is spent on the decode/encode (after fixing the initial bitEstimate)?

That is very silly... fixing that is a somewhat deeper change though.  I guess we'd need to
read the .tii file directly (not use SegmentTermEnum), and then copy the UTF8 bytes straight
without going through UTF16...

Do you have comparisons with pre-3.5 (before we cutover to this more RAM-efficient (but CPU
heavy on load) terms index)?  Probably that less CPU on init, but more RAM held for the lifetime
of the reader...?
                
> Improve load time of .tii files
> -------------------------------
>
>                 Key: LUCENE-3932
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3932
>             Project: Lucene - Java
>          Issue Type: Improvement
>    Affects Versions: 3.5
>         Environment: Linux
>            Reporter: Sean Bridges
>
> We have a large 50 gig index which is optimized as one segment, with a 66 MEG .tii file.
 This index has no norms, and no field cache.
> It takes about 5 seconds to load this index, profiling reveals that 60% of the time is
spent in GrowableWriter.set(index, value), and most of time in set(...) is spent resizing
PackedInts.Mutatable current.
> In the constructor for TermInfosReaderIndex, you initialize the writer with the line,
> {quote}GrowableWriter indexToTerms = new GrowableWriter(4, indexSize, false);{quote}
> For our index using four as the bit estimate results in 27 resizes.
> The last value in indexToTerms is going to be ~ tiiFileLength, and if instead you use,
> {quote}int bitEstimate = (int) Math.ceil(Math.log10(tiiFileLength) / Math.log10(2));
> GrowableWriter indexToTerms = new GrowableWriter(bitEstimate, indexSize, false);{quote}
> Load time improves to ~ 2 seconds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message