Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <61634103.340591268916087212.JavaMail.jira@brutus.apache.org>
Date: Thu, 18 Mar 2010 12:41:27 +0000 (UTC)
From: "Earwin Burrfoot (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-2328) IndexWriter.synced  field
 accumulates data leading to a Memory Leak
In-Reply-To: <1070322950.322111268856930600.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846880#action_12846880 ] 

Earwin Burrfoot commented on LUCENE-2328:
-----------------------------------------

> EG running merges (or any still-open files) should not be sync'd.
Files that are still being written should not be synced, that's kinda obvious.

> Not necessarily all closed files should be sync'd either - eg any files that were opened & closed while we were syncing (since syncing can take some time) should not then be sync'd.
This one is not so obvious.
I assume that on calling syncEveryoneAndHisDog() you should sync all files that have been written to, and were closed, and not yet deleted.

> Maybe we change Dir.sync to take a Collection<String>?
What does that alone give us over the current situation? You can call Dir.sync() repeatedly, it's all the same.

> Or... I wonder if calling sync on a file that's already been sync'd is really that wasteful... 
It can be on these systems, that just sync down everything. I don't believe in people writing good software : }

> IndexWriter.synced  field accumulates data leading to a Memory Leak
> -------------------------------------------------------------------
>
>                 Key: LUCENE-2328
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2328
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.9.1, 2.9.2, 3.0, 3.0.1
>         Environment: all
>            Reporter: Gregor Kaczor
>            Priority: Minor
>             Fix For: 3.1
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I am running into a strange OutOfMemoryError. My small test application does
> index and delete some few files. This is repeated for 60k times. Optimization
> is run from every 2k times a file is indexed. Index size is 50KB. I did analyze
> the HeapDumpFile and realized that IndexWriter.synced field occupied more than
> half of the heap. That field is a private HashSet without a getter. Its task is
> to hold files which have been synced already.
> There are two calls to addAll and one call to add on synced but no remove or
> clear throughout the lifecycle of the IndexWriter instance.
> According to the Eclipse Memory Analyzer synced contains 32618 entries which
> look like file names "_e065_1.del" or "_e067.cfs"
> The index directory contains 10 files only.
> I guess synced is holding obsolete data 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org