lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <j...@apache.org>
Subject [jira] Updated: (LUCENE-2295) Create a MaxFieldLengthAnalyzer to wrap any other Analyzer and provide the same functionality as MaxFieldLength provided on IndexWriter
Date Sat, 06 Mar 2010 14:02:27 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Uwe Schindler updated LUCENE-2295:
----------------------------------

    Attachment: LUCENE-2295.patch

Here is a first patch.

Robert & me found a bug in CharTokenizer that it not correctly sets the endOffset when
the underlying reader is not exhausted. This is fixed in the patch, too. The bug in CharTokenizer
was also there when sombody used the MaxFieldLength with IW -- possible highlighting problems
:-(

Nevertheless, CharTokenizer should be rewritten, the code is not understandable and ugly.
Too many states and branches and possibly uninitialized variables (i fixed by two asserts).

Deprecating IW.MaxFieldLength is not yet added, this is just the new Filter/Analyzer.

> Create a MaxFieldLengthAnalyzer to wrap any other Analyzer and provide the same functionality
as MaxFieldLength provided on IndexWriter
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-2295
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2295
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Shai Erera
>            Assignee: Uwe Schindler
>             Fix For: 3.1
>
>         Attachments: LUCENE-2295.patch
>
>
> A spinoff from LUCENE-2294. Instead of asking the user to specify on IndexWriter his
requested MFL limit, we can get rid of this setting entirely by providing an Analyzer which
will wrap any other Analyzer and its TokenStream with a TokenFilter that keeps track of the
number of tokens produced and stop when the limit has reached.
> This will remove any count tracking in IW's indexing, which is done even if I specified
UNLIMITED for MFL.
> Let's try to do it for 3.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message