lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramazan VARLIKLI (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2192) Memory Leak
Date Mon, 04 Jan 2010 18:47:55 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796286#action_12796286
] 

Ramazan VARLIKLI commented on LUCENE-2192:
------------------------------------------

no ,
The old _X.cfs files are removed correctly 
but _X.cfs file is growing in size . 

Actually  I tried to remove all _X.cfs files with java io commands but it didn't work. Lucene
is keeping everything in memory and adds new document to it . I just want to remind this problem
happens as long as within one JVM instance . If you shutdown it , it will start from scratch

> Memory Leak 
> ------------
>
>                 Key: LUCENE-2192
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2192
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.9
>            Reporter: Ramazan VARLIKLI
>
> Hi All ,
> I have been working on a problem with Lucene and now gave up after trying many different
possibilites which gives me a feeling that There is a bug on this .
> The scenario is we have an CMS applicaton into which we add new content every week ,
instead of updating the index which is a bit tricky, I prefer to delete all index documents
and add them again which is straightforward . The problem is Lucene doesn't delete the old
data somehow and increase the index size every time during the update . I also profile it
with java tools and see that even if I close the IndexWriter class and sent it to Garbage
Collector it holds all the docs in the memory .
> Here is the code I use 
> Directory directory = new SimpleFSDirectory(new File(path));
> writer = new IndexWriter(directory, analyzer, false,IndexWriter.MaxFieldLength.LIMITED);
> writer.deleteAll();
> //after adding docs close the indexwriter 
> writer.close();
> The above code invoked every time we need to update the index . I tried many different
scenario here to overcome the problem which includes physically removing the index directory(
see how desperate I am ) , optimizing , flushing, commiting indexwriter, create=true parameter
and so on . 
> Here is the index file size during creation. If I shutdown the application and restart
it , index size starts with 2,458 which is correct size.
> Any help will be appreciated
> _17.cfs   2,458 KB
> _18.cfs   3,990 KB
> _19.cfs  5,149 KB
> here is the Lucene logs during creationg of index files 3 times in a row 
> IFD [http-8080-1]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@6649
> IW 0 [http-8080-1]: setInfoStream: dir=org.apache.lucene.store.SimpleFSDirectory@C:\Documents
and Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
autoCommit=false mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@3b626c mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@baa6ba
ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 index=
> IW 0 [http-8080-1]: now flush at close
> IW 0 [http-8080-1]:   flush: segment=_17 docStoreSegment=_17 docStoreOffset=0 flushDocs=true
flushDeletes=true flushDocStores=true numDocs=2765 numBufDelTerms=0
> IW 0 [http-8080-1]:   index before flush 
> IW 0 [http-8080-1]: DW: flush postings as segment _17 numDocs=2765
> IW 0 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _17 numDocs=2765
> IW 0 [http-8080-1]: DW:   oldRAMSize=7485440 newFlushedSize=2472818 docs/MB=1,172.473
new/old=33.035%
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit = false]
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit = false]
> IFD [http-8080-1]: delete "_17.fdx"
> IFD [http-8080-1]: delete "_17.tis"
> IFD [http-8080-1]: delete "_17.frq"
> IFD [http-8080-1]: delete "_17.nrm"
> IFD [http-8080-1]: delete "_17.fdt"
> IFD [http-8080-1]: delete "_17.fnm"
> IFD [http-8080-1]: delete "_17.tii"
> IFD [http-8080-1]: delete "_17.prx"
> IFD [http-8080-1]: now checkpoint "segments_1j" [1 segments ; isCommit = false]
> IW 0 [http-8080-1]: LMP: findMerges: 1 segments
> IW 0 [http-8080-1]: LMP:   level 6.2247195 to 6.400742: 1 segments
> IW 0 [http-8080-1]: CMS: now merge
> IW 0 [http-8080-1]: CMS:   index: _17:c2765
> IW 0 [http-8080-1]: CMS:   no more merges pending; now return
> IW 0 [http-8080-1]: CMS: now merge
> IW 0 [http-8080-1]: CMS:   index: _17:c2765
> IW 0 [http-8080-1]: CMS:   no more merges pending; now return
> IW 0 [http-8080-1]: now call final commit()
> IW 0 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 0 [http-8080-1]: startCommit index=_17:c2765 changeCount=5
> IW 0 [http-8080-1]: now sync _17.cfs
> IW 0 [http-8080-1]: done all syncs
> IW 0 [http-8080-1]: commit: pendingCommit != null
> IW 0 [http-8080-1]: commit: wrote segments file "segments_1k"
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1j"
> IFD [http-8080-1]: delete "_16.cfs"
> IFD [http-8080-1]: delete "segments_1j"
> IW 0 [http-8080-1]: commit: done
> IW 0 [http-8080-1]: at close: _17:c2765
> IFD [http-8080-1]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@fb1ba7
> IW 1 [http-8080-1]: setInfoStream: dir=org.apache.lucene.store.SimpleFSDirectory@C:\Documents
and Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
autoCommit=false mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@1d49559 mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@1990e2d
ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 index=
> IW 1 [http-8080-1]: now flush at close
> IW 1 [http-8080-1]:   flush: segment=_18 docStoreSegment=_18 docStoreOffset=0 flushDocs=true
flushDeletes=true flushDocStores=true numDocs=2765 numBufDelTerms=0
> IW 1 [http-8080-1]:   index before flush 
> IW 1 [http-8080-1]: DW: flush postings as segment _18 numDocs=2765
> IW 1 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _18 numDocs=2765
> IW 1 [http-8080-1]: DW:   oldRAMSize=9517056 newFlushedSize=4042307 docs/MB=717.242 new/old=42.474%
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = false]
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = false]
> IFD [http-8080-1]: delete "_18.nrm"
> IFD [http-8080-1]: delete "_18.frq"
> IFD [http-8080-1]: delete "_18.fdx"
> IFD [http-8080-1]: delete "_18.tii"
> IFD [http-8080-1]: delete "_18.fdt"
> IFD [http-8080-1]: delete "_18.prx"
> IFD [http-8080-1]: delete "_18.fnm"
> IFD [http-8080-1]: delete "_18.tis"
> IFD [http-8080-1]: now checkpoint "segments_1k" [1 segments ; isCommit = false]
> IW 1 [http-8080-1]: LMP: findMerges: 1 segments
> IW 1 [http-8080-1]: LMP:   level 6.2247195 to 6.6112633: 1 segments
> IW 1 [http-8080-1]: CMS: now merge
> IW 1 [http-8080-1]: CMS:   index: _18:c2765
> IW 1 [http-8080-1]: CMS:   no more merges pending; now return
> IW 1 [http-8080-1]: CMS: now merge
> IW 1 [http-8080-1]: CMS:   index: _18:c2765
> IW 1 [http-8080-1]: CMS:   no more merges pending; now return
> IW 1 [http-8080-1]: now call final commit()
> IW 1 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 1 [http-8080-1]: startCommit index=_18:c2765 changeCount=5
> IW 1 [http-8080-1]: now sync _18.cfs
> IW 1 [http-8080-1]: done all syncs
> IW 1 [http-8080-1]: commit: pendingCommit != null
> IW 1 [http-8080-1]: commit: wrote segments file "segments_1l"
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1k"
> IFD [http-8080-1]: delete "segments_1k"
> IFD [http-8080-1]: delete "_17.cfs"
> IW 1 [http-8080-1]: commit: done
> IW 1 [http-8080-1]: at close: _18:c2765
> IFD [http-8080-1]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@7ceec1
> IW 2 [http-8080-1]: setInfoStream: dir=org.apache.lucene.store.SimpleFSDirectory@C:\Documents
and Settings\rvarlikli\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\Clipbank3.5\lucene
autoCommit=false mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@1edacc mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@1ae9ba8
ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 index=
> IW 2 [http-8080-1]: now flush at close
> IW 2 [http-8080-1]:   flush: segment=_19 docStoreSegment=_19 docStoreOffset=0 flushDocs=true
flushDeletes=true flushDocStores=true numDocs=2765 numBufDelTerms=0
> IW 2 [http-8080-1]:   index before flush 
> IW 2 [http-8080-1]: DW: flush postings as segment _19 numDocs=2765
> IW 2 [http-8080-1]: DW: closeDocStore: 2 files to flush to segment _19 numDocs=2765
> IW 2 [http-8080-1]: DW:   oldRAMSize=11188224 newFlushedSize=5229106 docs/MB=554.457
new/old=46.738%
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = false]
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = false]
> IFD [http-8080-1]: delete "_19.tis"
> IFD [http-8080-1]: delete "_19.prx"
> IFD [http-8080-1]: delete "_19.nrm"
> IFD [http-8080-1]: delete "_19.fnm"
> IFD [http-8080-1]: delete "_19.fdx"
> IFD [http-8080-1]: delete "_19.fdt"
> IFD [http-8080-1]: delete "_19.tii"
> IFD [http-8080-1]: delete "_19.frq"
> IFD [http-8080-1]: now checkpoint "segments_1l" [1 segments ; isCommit = false]
> IW 2 [http-8080-1]: LMP: findMerges: 1 segments
> IW 2 [http-8080-1]: LMP:   level 6.2247195 to 6.722014: 1 segments
> IW 2 [http-8080-1]: CMS: now merge
> IW 2 [http-8080-1]: CMS:   index: _19:c2765
> IW 2 [http-8080-1]: CMS:   no more merges pending; now return
> IW 2 [http-8080-1]: CMS: now merge
> IW 2 [http-8080-1]: CMS:   index: _19:c2765
> IW 2 [http-8080-1]: CMS:   no more merges pending; now return
> IW 2 [http-8080-1]: now call final commit()
> IW 2 [http-8080-1]: startCommit(): start sizeInBytes=0
> IW 2 [http-8080-1]: startCommit index=_19:c2765 changeCount=5
> IW 2 [http-8080-1]: now sync _19.cfs
> IW 2 [http-8080-1]: done all syncs
> IW 2 [http-8080-1]: commit: pendingCommit != null
> IW 2 [http-8080-1]: commit: wrote segments file "segments_1m"
> IFD [http-8080-1]: now checkpoint "segments_1m" [1 segments ; isCommit = true]
> IFD [http-8080-1]: deleteCommits: now decRef commit "segments_1l"
> IFD [http-8080-1]: delete "_18.cfs"
> IFD [http-8080-1]: delete "segments_1l"
> IW 2 [http-8080-1]: commit: done
> IW 2 [http-8080-1]: at close: _19:c2765

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message