lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Grant Ingersoll (JIRA)" <>
Subject [jira] Commented: (LUCENE-1058) New Analyzer for buffering tokens
Date Tue, 27 Nov 2007 20:32:43 GMT


Grant Ingersoll commented on LUCENE-1058:

Maybe I'm missing something?

No, I don't think you are missing anything in that use case, it's just an example of its use.
 And I am not totally sold on this approach, but mostly am :-) 

I had originally considered your option, but didn't feel it was satisfactory for the case
where you are extracting things like proper nouns or maybe it is generating a category value.
 The more general case is where not all the tokens are needed (in fact, very few are).  In
those cases, you have to go back through the whole list of cached tokens in order to extract
the ones you want.  In fact, thinking some more of on it, I am not sure my patch goes far
enough in the sense that what if you want it to buffer in mid stream.  

For example, if you had:
Proper Noun TF

and Proper Noun TF is solely responsible for setting aside proper nouns as it comes across
them in the stream.

As for the convoluted cross-field logic, I don't think it is all that convoluted.  There are
only two fields and the implementing Analyzer takes care of all of it.  Only real requirement
the application has is that the fields be ordered correctly.  

I do agree somewhat about the pre-analysis approach, except for the case where there may be
a large number of tokens in the source field, in which case, you are holding them around in
memory (maxFieldLength mitigates to some extent.)  Also, it puts the onus on the app. writer
to do it, when it could be pretty straight forward for Lucene to do it w/o it's usual analysis

At any rate, separate of the CollaboratingAnalyzer, I do think the CachedTokenFilter is useful,
especially in supporting the pre-analysis approach.

> New Analyzer for buffering tokens
> ---------------------------------
>                 Key: LUCENE-1058
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>            Reporter: Grant Ingersoll
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 2.3
>         Attachments: LUCENE-1058.patch, LUCENE-1058.patch, LUCENE-1058.patch, LUCENE-1058.patch,
> In some cases, it would be handy to have Analyzer/Tokenizer/TokenFilters that could siphon
off certain tokens and store them in a buffer to be used later in the processing pipeline.
> For example, if you want to have two fields, one lowercased and one not, but all the
other analysis is the same, then you could save off the tokens to be output for a different
> Patch to follow, but I am still not sure about a couple of things, mostly how it plays
with the new reuse API.
> See;#54397

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message