lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2380) Add FieldCache.getTermBytes, to load term data as byte[]
Date Wed, 07 Apr 2010 18:00:34 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854621#action_12854621
] 

Yonik Seeley commented on LUCENE-2380:
--------------------------------------

bq. We could also do shared byte[] blocks (private), with a public method to retrieve the
BytesRef for a given doc?

Absolutely!  Now that we are in control, it would be a crime not not share the byte[]
Seems like one should pass in a BytesRef to be filled in... that would be most efficient for
people doing simple stuff like compare docid1 to docid2.  Returning a reused BytesRef could
also work (as TermsEnum does) but it's less efficient for anything needing a state of more
than 1 BytesRef since it then requires copying.

We can further save space by putting the length as a vInt in the byte[] - most would be a
single byte.
Then we just need an int[] as an index into the byte[]... or potentially packed ints.

We'll also need an implementation that can span multiple byte[]s for larger than 2GB support.
 The correct byte[] to look into is then simply a function of the docid (as is done in Solr's
UnInvertedField).

We could possibly play games with the offsets into the byte[] too - encode as a delta against
the average instead of an absolute offset.  So offset = average_size * ord + get_delta(ord)

> Add FieldCache.getTermBytes, to load term data as byte[]
> --------------------------------------------------------
>
>                 Key: LUCENE-2380
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2380
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>             Fix For: 3.1
>
>
> With flex, a term is now an opaque byte[] (typically, utf8 encoded unicode string, but
not necessarily), so we need to push this up the search stack.
> FieldCache now has getStrings and getStringIndex; we need corresponding methods to load
terms as native byte[], since in general they may not be representable as String.  This should
be quite a bit more RAM efficient too, for US ascii content since each character would then
use 1 byte not 2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message