lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Winton Davies <wdav...@overture.com>
Subject RE: Indexing problem
Date Fri, 02 Nov 2001 21:22:36 GMT
Hi,

  I implemented this, and I do have a question --
  What should the MergeFactor and MaxMergeDocuments be ?
 

  Cheers,
   Winton


>Well, I don't know if there's an archive of the list, so this what Doug
>wrote:
>
>"
>A more efficient and slightly more complex approach would be to build
>large
>indexes in RAM, and copy them to disk with IndexWriter.addIndexes:
>  IndexWriter fsWriter = new IndexWriter(new File(...), analyzer, true);
>  while (... more docs to index...)
>    RAMDirectory ramDir = new RAMDirectory();
>    IndexWriter ramWriter = new IndexWriter(ramDir, analyzer, true);
>    ... add 100,000 docs to ramWriter ...
>    ramWriter.optimize();
>    ramWriter.close();
>    fsWriter.addIndexes(new Directory[] { ramDir });
>  }
>  fsWriter.optimize();
>  fsWriter.close();
>"
>
>Scott
>
>> -----Original Message-----
>> From: Paul Friedman [mailto:pfriedman@macromedia.com]
>> Sent: Friday, November 02, 2001 9:13 AM
>> To: 'Lucene Users List'
>> Subject: RE: Indexing problem
>>
>>
>> Where can I get Doug's example of indexing in memory and then
>> writing it out
>> to disk? I just recently subscribed to this list and I can't
>> find it in the
>> archive.
>>
>> Thanks.
>> Paul
>>
>> -----Original Message-----
>> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
>> Sent: Friday, November 02, 2001 7:14 AM
>> To: 'Lucene Users List'
>> Subject: RE: Indexing problem
>>
>>
>> Yes.  You have too many open files.  There are a few things
>> you can try.  1)
>> Increase the number of file handles your system has
>> available.  Yes, there
>> is a setting for this in Windows.   2) Make sure that you have the
>> IndexWriter.maxMergeDocs set to Integer.MAX_VALUE (the
>> default).  3) Try
>> smaller values for IndexWriter.mergeFactor (default is 10).
>> 4) When all
>> else fails, do all your indexing in memory and then write it
>> out to disk
>> when you're done.  Doug posted an example of this just a
>> couple days ago.
>>
>> Scott
>>
>> > -----Original Message-----
>> > From: Daryl Thachuk [mailto:daryl@montagetech.com]
>> > Sent: Thursday, November 01, 2001 7:46 PM
>> > To: lucene-user@jakarta.apache.org
>> > Subject: Indexing problem
>> >
>> >
>> > Hi
>> >
>> > Since upgrading to 1.2  we've started getting the following
>> > error when
>> > creating an index in a directory with a large amount of
>> > files. Previous
>> > versions of Lucene were quite happy to index this directory.
>> >
>> > Any thoughts as to the cause?
>> >
>> > -d
>> >
>> > java.io.FileNotFoundException:
>> > /private/Network/Servers/montage/Volumes/Disk2/Users/daryl/Lib
>> > rary/Index/index5.
>> > mtlibx/_n8.f3 (Too many open files)
>> > 	at java.io.RandomAccessFile.open(Native Method)
>> > 	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
>> > 	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:143)
>> > 	at
>> > org.apache.lucene.store.FSInputStream$Descriptor.<init>(Unknown
>> > Source)
>> > 	at org.apache.lucene.store.FSInputStream.<init>(Unknown Source)
>> > 	at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
>> > 	at
>> > org.apache.lucene.index.SegmentReader.openNorms(Unknown Source)
>> > 	at org.apache.lucene.index.SegmentReader.<init>(Unknown Source)
>> > 	at
>> > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
>> > 	at
>> > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
>> > 	at
>> > org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown
>> > Source)
>> > 	at
>> > org.apache.lucene.index.IndexWriter.addDocument(Unknown Source)
>> > 	at IndexCreator.indexDocs(IndexCreator.java:75)
>> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
>> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
>> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
>> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
>> > 	at IndexCreator.indexDocs(IndexCreator.java:67)
>> > 	at IndexCreator.createIndex(IndexCreator.java:44)
>> >
>> > ------
>> > Daryl Thachuk		daryl@montagetech.com
>> > Montage Technologies Inc.
>> > http://www.montagetech.com
>> >
>> >
>> > --
>> > To unsubscribe, e-mail:
>> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
>> For additional commands, e-mail:
>> <mailto:lucene-user-help@jakarta.apache.org>
>>
>> --
>> To unsubscribe, e-mail:
>> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
>> For additional commands, e-mail:
>> <mailto:lucene-user-help@jakarta.apache.org>
>>


--
To unsubscribe, e-mail:   <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Mime
View raw message