lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: Payload API
Date Mon, 19 Nov 2007 20:13:15 GMT

"Yonik Seeley" <yonik@apache.org> wrote:

> On Nov 19, 2007 2:03 PM, Michael McCandless <lucene@mikemccandless.com>
> wrote:
> > Maybe you mean that each Token must be fully independent because there
> > are plenty of filters that hold onto each Token long after next() is
> > called again, and then serve them up again later
> 
> Yes.

OK.

> > But this is why we have the Token next() method (which returns a
> > "stable" Token that the callee can't change later) vs the Token
> > next(Token) method which allows the callee to re-use the passed in
> > Token on the next call.
> 
> Right...  that works great for termBuffer, but the difference is that
> Token owns the termBuffer (contents are always copied into it).

Right.

> > So eg in TokenStream.next() base implementation we would have to make
> > a new copy of the payload byte[] and set it in the copied token.  This
> > way when a caller calls next() they get a full private copy, even of
> > the by-reference payload.
> 
> Perhaps I'm the one missing something, but it still doesn't seem like
> this works a filter can set the reference to the byte[] and is allowed
> to reuse it.  The particular filter setting the payload is in control,
> not next().

If the filter implements next() then that means it will never change
the byte[] payload that it set on the token -- the "private copy" API.

But if it implements next(Token) then it is allowed to change it on
the next call to next(Token) -- the "allow reuse" API.

So if a newish filter implements the "reuse" API, but an older filter
comes after it in the chain and calls next(), that older filter gets a
full copy because the default next() in TokenStream makes the copy.

I think?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message