lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2283) Possible Memory Leak in StoredFieldsWriter
Date Wed, 24 Feb 2010 14:48:28 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837811#action_12837811
] 

Michael McCandless commented on LUCENE-2283:
--------------------------------------------

bq. a yourkit snapshot showed that the PerDocs for an IndexWriter were using ~40M of memory

What was IW's ramBufferSizeMB when you saw this?

bq. however i have reports that eventually the memory is completely exhausted resulting in
out of memory errors.

Hmm, that makes me nervous, because I think in this case the use should be bounded.

bq. As a side note, closing the index writer at commit time would sometimes fail, resulting
in some following updates to fail because the index writer was locked and couldn't be reopened
until the old index writer was garbage collected, so i don't want to go back to this for commits.

That doesn't sound good!  Can you post some details on this (eg an exception)?

But, anyway, keeping the same IW open and just calling commit is (should be) fine.

bq. As far as a fix goes, wouldn't it be better to have the RAMFile's used for stored fields
pull and return byte buffers from the byte block pool on the DocumentsWriter?

Yes, that's a great solution -- a single pool.  But that's a somewhat bigger change.  I guess
we can pass a byte[] allocator to RAMFile.  It'd have to be a new pool, too (DW's byte blocks
are 32KB not the 1KB that RAMFile uses).

> Possible Memory Leak in StoredFieldsWriter
> ------------------------------------------
>
>                 Key: LUCENE-2283
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2283
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.4.1
>            Reporter: Tim Smith
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> StoredFieldsWriter creates a pool of PerDoc instances
> this pool will grow but never be reclaimed by any mechanism
> furthermore, each PerDoc instance contains a RAMFile.
> this RAMFile will also never be truncated (and will only ever grow) (as far as i can
tell)
> When feeding documents with large number of stored fields (or one large dominating stored
field) this can result in memory being consumed in the RAMFile but never reclaimed. Eventually,
each pooled PerDoc could grow very large, even if large documents are rare.
> Seems like there should be some attempt to reclaim memory from the PerDoc[] instance
pool (or otherwise limit the size of RAMFiles that are cached) etc

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message