lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@blackwell.co.uk>
Subject Re: Efficient document spooling and indexing
Date Thu, 22 Nov 2001 12:16:01 GMT
Data may not be committed to disk, buffers flushed, files
closed, etc. until IndexWriter.close() is called, but file
IO does happen before then.  So I would expect the answer
to your question to be no.


--
Ian.
ian.lea@blackwell.co.uk


Otis Gospodnetic wrote:
> 
> Hello,
> 
> This is from a thread from about 2 weeks ago.
> What is the answer to this question?
> If data is written to disk only when IndexWriter's close() is called,
> wouldn't the sample code below be as efficient as the sample code that
> uses RAMDirectory, further down?
> 
> Thanks,
> Otis
> 
> ----
> When using the FSWriter, the actual file io doesn't occur until I close
> the writer, right?  So wouldn't it be just as efficient to do the
> following:
> 
> IndexWriter fsWriter = new IndexWriter(new File(...), analyzer, false);
>   while (... more docs to index...)
>     ... add 100,000 docs to fsWriter ...
>   }
>   fsWriter.optimize();
>   fsWriter.close();
> 
> -----Original Message-----
> From: Scott Ganyo [mailto:scott.ganyo@eTapestry.com]
> Sent: Friday, November 02, 2001 10:47 AM
> To: 'Lucene Users List'
> Subject: RE: Indexing problem
> 
> Well, I don't know if there's an archive of the list, so this what Doug
> wrote: "
> A more efficient and slightly more complex approach would be to build
> large
> indexes in RAM, and copy them to disk with IndexWriter.addIndexes:
>   IndexWriter fsWriter = new IndexWriter(new File(...), analyzer,
> true);
>   while (... more docs to index...)
>     RAMDirectory ramDir = new RAMDirectory();
>     IndexWriter ramWriter = new IndexWriter(ramDir, analyzer, true);
>     ... add 100,000 docs to ramWriter ...
>     ramWriter.optimize();
>     ramWriter.close();
>     fsWriter.addIndexes(new Directory[] { ramDir });
>   }
>   fsWriter.optimize();
>   fsWriter.close();
> "
> 
> Scott

--
To unsubscribe, e-mail:   <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Mime
View raw message