lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-1859) TermAttributeImpl's buffer will never "shrink" if it grows too big
Date Wed, 26 Aug 2009 18:33:59 GMT


Uwe Schindler commented on LUCENE-1859:

The problem is, that it may be possible to shrink the buffer once per document, when TokenStream's
reset() is called (which is done before each new document). To achieve this, all TokenStreams
must notify the termattribute in reset() to shrink its size, which is impractical.

On the other hand, the reallocation would always be for each token (you call that inner loop).

I agree, that normally, the tokens will not grow very large (if they do, you do something
wrong during tokenization). Even things like KeywordTokenizer that only creates one token
has an upper limit of the term size (as far as I know).

I would set this to minor and would not take care before 2.9. The problem of maybe large buffers
was there even in older versions with Token as attribute implementation. It is the same problem
like preserving an ArrayList for very long time, it also only grows but never automatically

> TermAttributeImpl's buffer will never "shrink" if it grows too big
> ------------------------------------------------------------------
>                 Key: LUCENE-1859
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Analysis
>    Affects Versions: 2.9
>            Reporter: Tim Smith
> This was also an issue with Token previously as well
> If a TermAttributeImpl is populated with a very long buffer, it will never be able to
reclaim this memory
> Obviously, it can be argued that Tokenizer's should never emit "large" tokens, however
it seems that the TermAttributeImpl should have a reasonable static "MAX_BUFFER_SIZE" such
that if the term buffer grows bigger than this, it will shrink back down to this size once
the next token smaller than MAX_BUFFER_SIZE is set
> I don't think i have actually encountered issues with this yet, however it seems like
if you have multiple indexing threads, you could end up with a char[Integer.MAX_VALUE] per
thread (in the very worst case scenario)
> perhaps growTermBuffer should have the logic to shrink if the buffer is currently larger
than MAX_BUFFER_SIZE and it needs less than MAX_BUFFER_SIZE

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message