lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurabh Gokhale <saurabhgokh...@gmail.com>
Subject Re: Questions on index Writer
Date Sat, 16 Jul 2011 16:20:50 GMT
Hi Eric

Thanks for the reply.

*Index Code:*

Here is the code snippet that I am using for creating index writer.

indexWriter = new IndexWriter(FSDirectory.open(new File(indexDirName)),
getAnalyzer(), true, MaxFieldLength.UNLIMITED);

Does this statement cleans up existing index files? If yes, then how do I
tackle a scenario where lets say I brought down my application server
hosting code to create lucene index and then start it again, this way a new
JVM will start creating indexWriter instance again which will eventually
clean up my existing index files.

*Code Setup:*

I have a singleton IndexObjectsCreator factory class with indexWriter as an
instance level variable. This factory creates all the objects required like
writer, reader, searcher etc. I create factory instance and indexwriter
instance only once and returns the same instance to the client.

Client can eventually call close on the indexWriter. If it does, then
I create a new indexWriter and return back for the next call for the writer.
My factory instance remains singleton for the duration of the application
run.


*MayBeMerge:*
Yes, as per your suggestion, I will skip this method call, if the segment
merge process operates well within itself automatically then I dont see a
need to call this method as my intetions were to suggest a merge request to
lucene.


thanks again

Saurabh


On Fri, Jul 15, 2011 at 3:03 PM, Erick Erickson <erickerickson@gmail.com>wrote:

> Index files should not be disappearing unless you're using the form
> of opening an indexwriter that creates a new index. We'd need to see
> the code you use top open the IW to provide more help.
>
> If all you're doing is looking at the index directory, segments will
> disappear
> as they are merged so that's also what you might be seeing. Can you give
> us more details?
>
> A good time to commit? Well, that depends. It really depends upon your app.
> Given what you've described, and assuming that you've set reasonable
> numbers for, say, setMaxBufferedDocs, setRAMBufferSizeMB just letting the
> commit happen when you close is probably fine.
>
> What do you hope to accomplish by executing maybeMerge()? Personally I'd
> stay away from that unless you're solving a specific question and just let
> the
> segment merge process operate to handle all that...
>
> Best
> Erick
>
> On Fri, Jul 15, 2011 at 3:03 PM, Saurabh Gokhale
> <saurabhgokhale@gmail.com> wrote:
> > Hi All,
> >
> > I have following questions about lucene indexWriter. I am using version
> > 3.1.0.
> >
> > While indexing documents,
> > 1. When is the good time to commit changes? (indexWriter.commit) or just
> > close the writer after the indexing is done so that commit automatically
> > happens.
> > 2. When is the good time to merge indexes (indexWriter.maybeMerge()).  Is
> it
> > just before committing the changes or after indexing say X number of
> > documents. (I recently upgraded from 2.9.4 to 3.1 and I see 3.1 lucene
> > generates lot of small index files while indexing document)
> >
> > Also I have a problem where my lucene index files sometimes gets deleted
> > from the index folder. I am not sure what code snippet is causing the
> > existing index files to accidently get removed.
> >
> > My indexer runs in a thread loop where it indexes file whenever they are
> > available. When no more files are available, indexer thread closes the
> > writer and goes to sleep, after specific time, it again creates a new
> index
> > on the same folder and starts indexing new files if any available.
> >
> > A. Is it a wrong way to index files?
> > B. Because I close the index and open it again later, am I seeing my
> lucene
> > index files getting deleted?
> >
> > Thanks
> >
> > Saurabh
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message