lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Created: (LUCENE-2467) IndexWriter memory leak when large docs are indexed
Date Mon, 17 May 2010 16:06:45 GMT
IndexWriter memory leak when large docs are indexed
---------------------------------------------------

                 Key: LUCENE-2467
                 URL: https://issues.apache.org/jira/browse/LUCENE-2467
             Project: Lucene - Java
          Issue Type: Bug
          Components: Index
    Affects Versions: 3.0.1, 3.0, 2.9.2, 2.9.1, 2.9, 2.4.1, 2.4, 2.3.2, 2.3.1, 2.3, 2.3.3,
2.4.2, 2.9.3, 3.0.2, 3.1, 4.0
            Reporter: Michael McCandless
            Assignee: Michael McCandless
             Fix For: 2.9.3, 3.0.2, 3.1, 4.0


Spinoff from the java-user thread "IndexWriter and memory usage"...

IndexWriter has had a long standing memory leak, since LUCENE-843.

When the byte/char/int blocks are recycled to the common pool, the
per-thread DW classes incorrectly still hold a reference to them.

This normally is not a problem, since these buffers will be re-used
again.

But, if you index a massive document, causing IW to allocate more than
the RAM buffer allocated to it, then the leak happens.  So you could
have a 16 MB RAM buffer set, but if a huge doc required allocation of
200 MB worth of arrays, those 200 MB are never freed (well, until you
close the IW and deref it from the app).

It's even worse if you use multiple threads: if each thread has ever
had to index a massive document, then that thread incorrectly holds
onto the extra arrays.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message