lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Weiss (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3003) Move UnInvertedField into Lucene core
Date Thu, 31 Mar 2011 07:20:06 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013869#comment-13013869
] 

Dawid Weiss commented on LUCENE-3003:
-------------------------------------

For what it's worth, the instrumentation interface allows one to get exact allocation sizes
of objects. I put together a small spike at https://github.com/dweiss/poligon/tree/master/instrumenter
that measures the actual allocation size of byte[]. On my hotspot, 64-bit, this yields:

{noformat}
byte[0] takes 24 bytes.
byte[1] takes 32 bytes.
byte[2] takes 32 bytes.
byte[3] takes 32 bytes.
byte[4] takes 32 bytes.
byte[5] takes 32 bytes.
byte[6] takes 32 bytes.
byte[7] takes 32 bytes.
byte[8] takes 32 bytes.
byte[9] takes 40 bytes.
byte[10] takes 40 bytes.
byte[11] takes 40 bytes.
...
{noformat}

IBM's VM yields the same (64-bit), but the version of jrockit that I have (which may be an
old one, but is 64-bit!) yields:

{noformat}
byte[0] takes 16 bytes.
byte[1] takes 24 bytes.
byte[2] takes 24 bytes.
byte[3] takes 24 bytes.
byte[4] takes 24 bytes.
byte[5] takes 24 bytes.
byte[6] takes 24 bytes.
byte[7] takes 24 bytes.
byte[8] takes 24 bytes.
byte[9] takes 32 bytes.
byte[10] takes 32 bytes.
byte[11] takes 32 bytes.
byte[12] takes 32 bytes.
byte[13] takes 32 bytes.
byte[14] takes 32 bytes.
byte[15] takes 32 bytes.
byte[16] takes 32 bytes.
byte[17] takes 40 bytes.
{noformat}

Don't have access to a 32-bit system right now, but if you're keen on checking, checkout that
github repo and run:

{noformat}
cd instrumenter
mvn package
java -javaagent:target/instrumenter-0.1.0-SNAPSHOT.jar -version
{noformat}

> Move UnInvertedField into Lucene core
> -------------------------------------
>
>                 Key: LUCENE-3003
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3003
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 3.2, 4.0
>
>         Attachments: LUCENE-3003.patch, LUCENE-3003.patch
>
>
> Solr's UnInvertedField lets you quickly lookup all terms ords for a
> given doc/field.
> Like, FieldCache, it inverts the index to produce this, and creates a
> RAM-resident data structure holding the bits; but, unlike FieldCache,
> it can handle multiple values per doc, and, it does not hold the term
> bytes in RAM.  Rather, it holds only term ords, and then uses
> TermsEnum to resolve ord -> term.
> This is great eg for faceting, where you want to use int ords for all
> of your counting, and then only at the end you need to resolve the
> "top N" ords to their text.
> I think this is a useful core functionality, and we should move most
> of it into Lucene's core.  It's a good complement to FieldCache.  For
> this first baby step, I just move it into core and refactor Solr's
> usage of it.
> After this, as separate issues, I think there are some things we could
> explore/improve:
>   * The first-pass that allocates lots of tiny byte[] looks like it
>     could be inefficient.  Maybe we could use the byte slices from the
>     indexer for this...
>   * We can improve the RAM efficiency of the TermIndex: if the codec
>     supports ords, and we are operating on one segment, we should just
>     use it.  If not, we can use a more RAM-efficient data structure,
>     eg an FST mapping to the ord.
>   * We may be able to improve on the main byte[] representation by
>     using packed ints instead of delta-vInt?
>   * Eventually we should fold this ability into docvalues, ie we'd
>     write the byte[] image at indexing time, and then loading would be
>     fast, instead of uninverting

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message