lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Earwin Burrfoot (JIRA)" <>
Subject [jira] Commented: (LUCENE-2328) IndexWriter.synced field accumulates data leading to a Memory Leak
Date Thu, 18 Mar 2010 18:30:27 GMT


Earwin Burrfoot commented on LUCENE-2328:

> Must the Dir insist the file is closed in order to sync it?
Well, no, this can be relaxed.
Because default Directory clients - IW+IR will never call sync() on a file they didn't close
Also this client behaviour is guaranteed with current implementation - if someone calls current
sync() on an open file, it will fail on 'new RandomAccessFile'?

Currently, if someone subclasses FSDir, he already always needs to call initOutput(name) before
creating IndexOutput. This class is obviously not designed for easy extension :)
Plus, someone might extend FSDir to change syncing behaviour, so we should allow this and
not force people?

My hands are itching to do minor cleanups on classes touched by the patch, like moving fields
to the beginning of the class, so you damn know what it contains from the first glance, declaring
most of said fields final (will help with "Sync on non-final field" warnings too), replacing
HEX_DIGITS voodoo with Integer.toHexString, moving createIndexOutput to FSDir, as it is a
copypasted in all three child classes and has ugly "always call initOutput" comment.
Is that acceptable or should be rolled into another issue?

> IndexWriter.synced  field accumulates data leading to a Memory Leak
> -------------------------------------------------------------------
>                 Key: LUCENE-2328
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.9.1, 2.9.2, 3.0, 3.0.1
>         Environment: all
>            Reporter: Gregor Kaczor
>            Priority: Minor
>             Fix For: 3.1
>   Original Estimate: 1h
>  Remaining Estimate: 1h
> I am running into a strange OutOfMemoryError. My small test application does
> index and delete some few files. This is repeated for 60k times. Optimization
> is run from every 2k times a file is indexed. Index size is 50KB. I did analyze
> the HeapDumpFile and realized that IndexWriter.synced field occupied more than
> half of the heap. That field is a private HashSet without a getter. Its task is
> to hold files which have been synced already.
> There are two calls to addAll and one call to add on synced but no remove or
> clear throughout the lifecycle of the IndexWriter instance.
> According to the Eclipse Memory Analyzer synced contains 32618 entries which
> look like file names "_e065_1.del" or "_e067.cfs"
> The index directory contains 10 files only.
> I guess synced is holding obsolete data 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message