lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Chow (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-6814) PatternTokenizer indefinitely holds heap equal to max document it has ever tokenized
Date Thu, 24 Sep 2015 21:21:04 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907054#comment-14907054
] 

Alex Chow commented on LUCENE-6814:
-----------------------------------

bq. Hmm, how many {{PatternTokenizer}} instances do you have? With 24 bulk indexing threads
you should (I think?) only have at most 24 instances * 4 MB max should be ~100 MB.

{{PatternTokenizer}} instances also scale with the number of indices (technically shards?)
if you use your own analyzer based on a {{PatternAnalyzer}}. Custom analyzers get instantiated
per index since they have their own mappings. We have 168 indices, so that's a _lot_. One
heap dump I was poking around had 3432 instances, but it'd probably have grown more since
some analyzers haven't had to create their {{TokenStreamComponents}} yet.

> PatternTokenizer indefinitely holds heap equal to max document it has ever tokenized
> ------------------------------------------------------------------------------------
>
>                 Key: LUCENE-6814
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6814
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: Trunk, 5.4
>
>         Attachments: LUCENE-6814.patch, LUCENE-6814.patch
>
>
> Caught by Alex Chow in this Elasticsearch issue: https://github.com/elastic/elasticsearch/issues/13721
> Today, PatternTokenizer reuses a single StringBuilder, but it doesn't free its heap usage
after tokenizing is done.  We can either stop reusing, or ask it to {{.trimToSize}} when we
are done ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message