lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <yo...@apache.org>
Subject Re: Token termBuffer issues
Date Fri, 20 Jul 2007 18:52:42 GMT
On 7/19/07, Michael McCandless <lucene@mikemccandless.com> wrote:
> "Yonik Seeley" <yonik@apache.org> wrote:
> > I had previously missed the changes to Token that add support for
> > using an array (termBuffer):
> >
> > +  // For better indexing speed, use termBuffer (and
> > +  // termBufferOffset/termBufferLength) instead of termText
> > +  // to save new'ing a String per token
> > +  char[] termBuffer;
> > +  int termBufferOffset;
> > +  int termBufferLength;
> >
> > While I think this approach would have been best to start off with
> > rather than String,
> > I'm concerned that it will do little more than add overhead at this
> > point, resulting in slower code, not faster.
> >
> > - If any tokenizer or token filter tries setting the termBuffer, any
> > downstream components would need to check for both.  It could be made
> > backward compatible by constructing a string on demand, but that will
> > really slow things down, unless the whole chain is converted to only
> > using the char[] somehow.
>
> Good point: if your analyzer/tokenizer produces char[] tokens then
> your downstream filters would have to accept char[] tokens.
>
> I think on-demand constructing a String (and saving it as termText)
> would be an OK solution?  Why would that be slower than having to make
> a String in the first place (if we didn't have the char[] API)?  It's
> at least graceful degradation.

It's the rule rather than the exception though.  Pretty much
everything is based on String.

> > - It doesn't look like the indexing code currently pays any attention
> > to the char[], right?
>
> It does, in DocumentsWriter.addPosition().

Ah, thanks.

> > - What if both the String and char[] are set?  A filter that doesn't
> > know better sets the String... this doesn't clear the char[]
> > currently, should it?
>
> Currently the char[] wins, but good point: seems like each setter
> should null out the other one?

Certainly the String setter should null the char[] (that's the only
way to keep back compatibility), and probably vice-versa.

Note that there are many existing filters that directly access and
manipulate the package protected String termText.  These will need to
be changed.

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message