Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <21E33FF4-77B7-4AA9-AE7F-4C22D2737DC9@apache.org>
From: Grant Ingersoll <gsingers@apache.org>
To: java-dev@lucene.apache.org
In-Reply-To: <786fde50811130714i2c6fd9a9jafb51cf2c26c0175@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-6320--824012916
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: Add Token.copyInto(Token) API
Date: Thu, 13 Nov 2008 10:48:10 -0500
References: <786fde50811121051q21d13d4fmef4cbbb3ab2aec3c@mail.gmail.com>
 <A031B264-19DF-4FD4-8B00-094270BDF92C@apache.org>
 <786fde50811130714i2c6fd9a9jafb51cf2c26c0175@mail.gmail.com>

--Apple-Mail-6320--824012916
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Don't want to discourage you from contributing it, just suggesting you  
may want to make sure the people working on that patch are aware of  
the issue such that maybe it can be addressed.


On Nov 13, 2008, at 10:14 AM, Shai Erera wrote:

> Thanks. I am aware of this thread. Indeed it will change the way  
> TokenStreams are handled, and so copying a Token may not be  
> necessary. However, I can't tell now whether this won't be necessary  
> - I guess I'll just have to wait until it's out and I start using  
> it :-)
>
> Anyway, I've implemented it for myself, and thought this might be a  
> nice contribution. I can live without it in Lucene :-)
>
> Thanks
> Shai
>
> On Wed, Nov 12, 2008 at 10:02 PM, Grant Ingersoll  
> <gsingers@apache.org> wrote:
> Are you aware of LUCENE-1422?  There is likely going to be a new way  
> of dealing w/ TokenStreams all together, so you might want to have a  
> look there before continuing.
>
>
> On Nov 12, 2008, at 1:51 PM, Shai Erera wrote:
>
> Hi,
>
> I was thinking about adding a copyInto method to Token. The only way  
> to clone a token is by using its clone() or clone(char[], int, int,  
> int, int) methods. Both do the job, but allocate a Token instance.  
> While in 2.4 a Token constructor may actually get a char[] as input  
> (thus saving a char[] allocation), but it still allocates an instance.
>
> Even though the instance allocation is not that expensive, it does  
> allocate additional things, like String for the type, Payload and  
> String (for the text, even though that will be removed in 3.0).
> If an application wishes to keep one instance of Token around, and  
> copy into it other Tokens, it can call various methods to achieve  
> that, like setTermBuffer, setOffset etc. A copyInto is just a  
> convenient method for doing that.
>
> If you wonder about the use case, then here it is: I know that it's  
> advised to reuse the same Token instance in the TokenStream API  
> (basically make sure to call next(Token). But there might be  
> TokenFilters which will need to save a certain occurrance of a  
> token, do some processing and return it later. A good example is  
> StemmingFilter. One can think of such a filter to return the  
> original token in addition to the stemmed token (for examle, for the  
> word "tokens" in English, it will return "tokens" [original] and  
> "token" [stem]). In that case, the filter has to save the word  
> "tokens" so that it returns "tokens" first (or the stem, the order  
> does not matter) and next time its next(Token) is called, it should  
> return the stem (or original), before comsuming the next token from  
> the TokenStream.
>
> Anyway, I hope it's clear enough, but if not I can elaborate.
> If you think a copyInto() is worth the effort, I can quickly create  
> a patch for it).
>
> Shai
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


--Apple-Mail-6320--824012916
Content-Type: text/html;
	charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

<html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; ">Don't want to discourage you =
from contributing it, just suggesting you may want to make sure the =
people working on that patch are aware of the issue such that maybe it =
can be addressed.<div><br></div><div><br><div><div>On Nov 13, 2008, at =
10:14 AM, Shai Erera wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div =
dir=3D"ltr">Thanks. I am aware of this thread. Indeed it will change the =
way TokenStreams are handled, and so copying a Token may not be =
necessary. However, I can't tell now whether this won't be necessary - I =
guess I'll just have to wait until it's out and I start using it :-)<br> =
<br>Anyway, I've implemented it for myself, and thought this might be a =
nice contribution. I can live without it in Lucene =
:-)<br><br>Thanks<br>Shai<br><br><div class=3D"gmail_quote">On Wed, Nov =
12, 2008 at 10:02 PM, Grant Ingersoll <span dir=3D"ltr">&lt;<a =
href=3D"mailto:gsingers@apache.org">gsingers@apache.org</a>></span> =
wrote:<br> <blockquote class=3D"gmail_quote" style=3D"border-left: 1px =
solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: =
1ex;">Are you aware of LUCENE-1422? &nbsp;There is likely going to be a =
new way of dealing w/ TokenStreams all together, so you might want to =
have a look there before continuing.<div> <div></div><div =
class=3D"Wj3C7c"><br> <br> On Nov 12, 2008, at 1:51 PM, Shai Erera =
wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"border-left: =
1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: =
1ex;"> Hi,<br> <br> I was thinking about adding a copyInto method to =
Token. The only way to clone a token is by using its clone() or =
clone(char[], int, int, int, int) methods. Both do the job, but allocate =
a Token instance. While in 2.4 a Token constructor may actually get a =
char[] as input (thus saving a char[] allocation), but it still =
allocates an instance.<br> <br> Even though the instance allocation is =
not that expensive, it does allocate additional things, like String for =
the type, Payload and String (for the text, even though that will be =
removed in 3.0).<br> If an application wishes to keep one instance of =
Token around, and copy into it other Tokens, it can call various methods =
to achieve that, like setTermBuffer, setOffset etc. A copyInto is just a =
convenient method for doing that.<br> <br> If you wonder about the use =
case, then here it is: I know that it's advised to reuse the same Token =
instance in the TokenStream API (basically make sure to call =
next(Token). But there might be TokenFilters which will need to save a =
certain occurrance of a token, do some processing and return it later. A =
good example is StemmingFilter. One can think of such a filter to return =
the original token in addition to the stemmed token (for examle, for the =
word "tokens" in English, it will return "tokens" [original] and "token" =
[stem]). In that case, the filter has to save the word "tokens" so that =
it returns "tokens" first (or the stem, the order does not matter) and =
next time its next(Token) is called, it should return the stem (or =
original), before comsuming the next token from the TokenStream.<br> =
<br> Anyway, I hope it's clear enough, but if not I can elaborate.<br> =
If you think a copyInto() is worth the effort, I can quickly create a =
patch for it).<br> <br> Shai<br> </blockquote> <br> <br> =
<br></div></div> =
---------------------------------------------------------------------<br> =
To unsubscribe, e-mail: <a =
href=3D"mailto:java-dev-unsubscribe@lucene.apache.org" =
target=3D"_blank">java-dev-unsubscribe@lucene.apache.org</a><br> For =
additional commands, e-mail: <a =
href=3D"mailto:java-dev-help@lucene.apache.org" =
target=3D"_blank">java-dev-help@lucene.apache.org</a><br> <br> =
</blockquote></div><br></div></blockquote></div><br></div></body></html>=

--Apple-Mail-6320--824012916--