lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2295) Create a MaxFieldLengthAnalyzer to wrap any other Analyzer and provide the same functionality as MaxFieldLength provided on IndexWriter
Date Sun, 07 Mar 2010 09:50:27 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842384#action_12842384
] 

Uwe Schindler commented on LUCENE-2295:
---------------------------------------

Further investigantions showed, that there is some difference between using this filter/analyzer
and the current setting in IndexWriter. IndexWriter uses the given MaxFieldLength as maximum
value for all instances of the same field name. So if you add 100 fields "foo" (with each
1,000 terms) and have the default of 10,000 tokens, DocInverter will index 10 of these field
instances (10,000 terms in total) and the rest will be supressed.

If you use the Filter, the limit is per TokenStream, so the above example will index all field
instances and produce 100,000 terms.

But the current IndexWriter code has a bug, too: The check for too many terms is done after
the first token of each input stream is indexed, so in the abovce example, IW will index 10,089
terms, because once the limit is reached, each stream left will index one term. This could
be fixed (if really needed, as the MaxFieldLength in IW should be deprecated) by moving the
check up and dont even try to index the field and create the TokenStream.

I just wanted to add this difference here for further discussing.

> Create a MaxFieldLengthAnalyzer to wrap any other Analyzer and provide the same functionality
as MaxFieldLength provided on IndexWriter
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-2295
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2295
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Shai Erera
>            Assignee: Uwe Schindler
>             Fix For: 3.1
>
>         Attachments: LUCENE-2295.patch
>
>
> A spinoff from LUCENE-2294. Instead of asking the user to specify on IndexWriter his
requested MFL limit, we can get rid of this setting entirely by providing an Analyzer which
will wrap any other Analyzer and its TokenStream with a TokenFilter that keeps track of the
number of tokens produced and stop when the limit has reached.
> This will remove any count tracking in IW's indexing, which is done even if I specified
UNLIMITED for MFL.
> Let's try to do it for 3.1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message