lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Ganyo <scott.ga...@eTapestry.com>
Subject RE: Indexing problem
Date Fri, 02 Nov 2001 15:47:18 GMT
Well, I don't know if there's an archive of the list, so this what Doug
wrote:

"
A more efficient and slightly more complex approach would be to build large
indexes in RAM, and copy them to disk with IndexWriter.addIndexes:
  IndexWriter fsWriter = new IndexWriter(new File(...), analyzer, true);
  while (... more docs to index...)
    RAMDirectory ramDir = new RAMDirectory();
    IndexWriter ramWriter = new IndexWriter(ramDir, analyzer, true);
    ... add 100,000 docs to ramWriter ...
    ramWriter.optimize();
    ramWriter.close();
    fsWriter.addIndexes(new Directory[] { ramDir });
  }
  fsWriter.optimize();
  fsWriter.close();
"

Scott

> -----Original Message-----
> From: Paul Friedman [mailto:pfriedman@macromedia.com]
> Sent: Friday, November 02, 2001 9:13 AM
> To: 'Lucene Users List'
> Subject: RE: Indexing problem
> 
> 
> Where can I get Doug's example of indexing in memory and then 
> writing it out
> to disk? I just recently subscribed to this list and I can't 
> find it in the
> archive.  
> 
> Thanks.
> Paul
> 
> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Friday, November 02, 2001 7:14 AM
> To: 'Lucene Users List'
> Subject: RE: Indexing problem
> 
> 
> Yes.  You have too many open files.  There are a few things 
> you can try.  1)
> Increase the number of file handles your system has 
> available.  Yes, there
> is a setting for this in Windows.   2) Make sure that you have the
> IndexWriter.maxMergeDocs set to Integer.MAX_VALUE (the 
> default).  3) Try
> smaller values for IndexWriter.mergeFactor (default is 10).  
> 4) When all
> else fails, do all your indexing in memory and then write it 
> out to disk
> when you're done.  Doug posted an example of this just a 
> couple days ago.
> 
> Scott
> 
> > -----Original Message-----
> > From: Daryl Thachuk [mailto:daryl@montagetech.com]
> > Sent: Thursday, November 01, 2001 7:46 PM
> > To: lucene-user@jakarta.apache.org
> > Subject: Indexing problem
> > 
> > 
> > Hi
> > 
> > Since upgrading to 1.2  we've started getting the following 
> > error when 
> > creating an index in a directory with a large amount of 
> > files. Previous 
> > versions of Lucene were quite happy to index this directory.
> > 
> > Any thoughts as to the cause?
> > 
> > -d
> > 
> > java.io.FileNotFoundException: 
> > /private/Network/Servers/montage/Volumes/Disk2/Users/daryl/Lib
> > rary/Index/index5.
> > mtlibx/_n8.f3 (Too many open files)
> > 	at java.io.RandomAccessFile.open(Native Method)
> > 	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> > 	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:143)
> > 	at 
> > org.apache.lucene.store.FSInputStream$Descriptor.<init>(Unknown 
> > Source)
> > 	at org.apache.lucene.store.FSInputStream.<init>(Unknown Source)
> > 	at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > 	at 
> > org.apache.lucene.index.SegmentReader.openNorms(Unknown Source)
> > 	at org.apache.lucene.index.SegmentReader.<init>(Unknown Source)
> > 	at 
> > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
> > 	at 
> > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
> > 	at 
> > org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown 
> > Source)
> > 	at 
> > org.apache.lucene.index.IndexWriter.addDocument(Unknown Source)
> > 	at IndexCreator.indexDocs(IndexCreator.java:75)
> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
> > 	at IndexCreator.createIndex(IndexCreator.java:44)
> > 
> > ------
> > Daryl Thachuk		daryl@montagetech.com
> > Montage Technologies Inc.
> > http://www.montagetech.com
> > 
> > 
> > --
> > To unsubscribe, e-mail:   
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-user-help@jakarta.apache.org>
> 
> --
> To unsubscribe, e-mail:   
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail: 
> <mailto:lucene-user-help@jakarta.apache.org>
> 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message