lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 32847] New: - IndexWriter.addIndexes results in java.lang.OutOfMemoryError
Date Sun, 26 Dec 2004 22:36:45 GMT
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=32847>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=32847

           Summary: IndexWriter.addIndexes results in
                    java.lang.OutOfMemoryError
           Product: Lucene
           Version: 1.4
          Platform: Other
        OS/Version: Linux
            Status: NEW
          Severity: critical
          Priority: P2
         Component: Index
        AssignedTo: lucene-dev@jakarta.apache.org
        ReportedBy: ddreon1@yahoo.com


I'm re-opening a bug I logged previously. My previous bug report has 
disappeared. 

Issue: IndexWriter.addIndexes results in java.lang.OutOfMemoryError for large 
merges.

Until this writing, I've been merging successfully only through repetition, 
i.e. I keep repeating merges until a success. As my index size has grown, my 
success rate has steadily declined. I've reached the point where merges now 
fail 100% of the time. I can't merge.

My tests indicate the threshold is ~30GB on P4/800MB VM with 6 indexes. I have 
repeated my tests on many different machines (not machine dependent). I have 
repeated my test using local and attached storage devices (not storage 
dependent).

For what its worth, I believe the exception occurs entirely during the optimize 
process which is called implicitly after the merge. I say this because each 
time it appears the correct amount of bytes are written to the new index. Is it 
possible to decouple the merge and optimize processes?


The code snippet follows. I can send you the class file and 120GB data set. Let 
me know how you want it.

>>>>> code sample >>>>>

Directory[] sources = new Directory[paths.length];
...

Directory dest = FSDirectory.getDirectory( path, true);
IndexWriter writer = new IndexWriter( dest, new TermAnalyzer( 
StopWords.SEARCH_MAP), true);

writer.addIndexes( sources);
writer.close();

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message