lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: weird behavior of IndexReader.indexExists()
Date Tue, 09 May 2006 15:10:33 GMT
Somebody (ah, Andy) already mentioned that you may be getting unlucky and calling that IndexReader.indexExists
method right when the 'segments' file is being renamed.  It looks like there is no lock in
that method, but it looks like we may have to add it.

Take a look at the IndexReader.open(....) method to see how the Lock class is used.  Add it
to IndexReader.indexExists method(s) and see if you still get the error.  If you don't, please
put the patch with your changes in a new JIRA issues, so we can fix the problem.

Thanks,
Otis


----- Original Message ----
From: wenjie zheng <wenjie.zheng@gmail.com>
To: java-dev@lucene.apache.org
Sent: Tuesday, May 9, 2006 10:57:05 AM
Subject: Re: weird behavior of IndexReader.indexExists()

I am a little bit confused here.
Since I didn't change the lock directory or disable locking and it is on the
single server sharing same fs, does it mean I am not supposed to get the
error?
But I did get the error. What should I do?

Wenjie
On 5/9/06, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
>
> If you don't explicitly change the lock directory and do not disable
> locking, the same directory should be used.  I'm assuming this is all done
> on a single server sharing the same file system.  The locks are stored in
> the system's default temporary directory.  That's typically /tmp under
> UNIX/OSX and C:\WINDOWS\TEMP (I think) on Winblows.
>
> Otis
>
> ----- Original Message ----
> From: wenjie zheng <wenjie.zheng@gmail.com>
> To: java-dev@lucene.apache.org
> Sent: Tuesday, May 9, 2006 9:09:47 AM
> Subject: Re: weird behavior of IndexReader.indexExists()
>
> Thanks,
> I guess my question is how to make sure both instances are using the same
> lock directory.
>
> Wenjie
>
> On 5/9/06, Vanlerberghe, Luc <Luc.Vanlerberghe@bvdep.com> wrote:
> >
> > Make sure both instances are using the same lock directory.
> > The segments file should only be read or written while holding the
> > commit lock.
> >
> > If the lock directories don't match, you'll get more 'strange' errors...
> >
> > In Lucene 1.4.2 some methods did not use the lock, this has been patched
> > a couple of months ago.
> >
> > Luc
> >
> > -----Original Message-----
> > From: Andy Hind [mailto:andy.hind@alfresco.org]
> > Sent: dinsdag 9 mei 2006 13:47
> > To: java-dev@lucene.apache.org
> > Subject: RE: weird behavior of IndexReader.indexExists()
> >
> > Hi
> >
> > I think I have discovered this too.
> > It is on my list of issues to raise ....
> >
> > The index exist test looks for the segment file.
> > When the index is committing, and you are unlucky, this file may not be
> > found as the new segments file replaces the old one. The result is the
> > index appears not to exist.
> >
> > Regards
> >
> > Andy
> >
> > -----Original Message-----
> > From: wenjie zheng [mailto:wenjie.zheng@gmail.com]
> > Sent: 08 May 2006 18:57
> > To: java-dev@lucene.apache.org
> > Subject: Re: weird behavior of IndexReader.indexExists()
> >
> > This happens sometimes when number of docs is over 2000. So it's kinda
> > of
> > random.
> >
> > Wenjie
> >
> > On 5/8/06, wenjie zheng <wenjie.zheng@gmail.com> wrote:
> > >
> > > I created an index with more than 30,000 text files.
> > > I used indexExists() to determine either to create a new index or to
> > add
> > > docs to the existing index.
> > >
> > > But when the num of docs in the index was over 3,000 (sometimes 3,400,
> > > sometimes 3,200), the indexExists function returns false, so I ended
> > up
> > > recreating a new index.
> > >
> > > Here is my code:
> > > if index exists, we will add files to it, otherwise, create a new
> > index.
> > > In either case, an IndexingThread will be spawn to do that.
> > >          if(IndexReader.indexExists(indexDir)){
> > >                 logger.info("Working on existing index ...");
> > >                 IndexingThread.startIndexingThread(Username, new
> > > File(propsFile), new File(indexDir), docs,
> > >                               new StandardAnalyzer(), false);
> > >           }else{
> > >                 logger.info("Create a new index ...");
> > >                 IndexingThread.startIndexingThread(Username, new
> > > File(propsFile), new File(indexDir), docs,
> > >                                new StandardAnalyzer(), true);
> > >            }
> > >
> > >
> > > inside the startIndexingThread function, I am calling the following
> > > function to add files to the index:
> > >     /**
> > >      * Add an array of Files to an index
> > >      *
> > >      * @param propsFile the properties file
> > >      * @param indexDir  the folder where index files will be created
> > in
> > >      * @param docs      an array of Files to be add to the index
> > >      * @param analyzer  any Analyzer object
> > >      */
> > >     public void addFiles(File propsFile, File indexDir, File[] docs,
> > > Analyzer analyzer, boolean overwrite) throws Exception {
> > >         Properties props = new Properties(new
> > FileInputStream(propsFile));
> > >
> > >         if(overwrite || IndexReader.indexExists(indexDir)){ //either
> > > overwrite or working on an existing index
> > >             Directory index = FSDirectory.getDirectory(indexDir,
> > > overwrite);
> > >             IndexWriter writer = new IndexWriter(index, analyzer,
> > > overwrite);
> > >
> > >             FileIndexer indexer = new FileIndexer(props);
> > >
> > >             long start = new Date().getTime();
> > >             indexer.index(writer, docs);
> > >             writer.optimize();
> > >             writer.close ();     // close the writer
> > >             index.close();      // close the index Directory
> > >             long end = new Date().getTime();
> > >
> > >             logger.info("Total time: " + (end - start) + " ms");
> > >
> > >         }else{
> > >             logger.error("Index files are not found: " +
> > > indexDir.getAbsolutePath() + ", overwrite = false");
> > >         }
> > >     }
> > >
> > > Thanks,
> > > Wenjie
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>




---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message