lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wenjie zheng" <wzh...@webmail.us>
Subject Re: weird behavior of IndexReader.indexExists()
Date Wed, 17 May 2006 15:09:19 GMT
I put Lock in IndexReader.indexExists function, and testes for a few days
It worked fine. I never had that mistery problem.

How can put the patch in a JIRA issue?

Thanks,
Wenjie
On 5/9/06, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
>
> Somebody (ah, Andy) already mentioned that you may be getting unlucky and
> calling that IndexReader.indexExists method right when the 'segments' file
> is being renamed.  It looks like there is no lock in that method, but it
> looks like we may have to add it.
>
> Take a look at the IndexReader.open(....) method to see how the Lock class
> is used.  Add it to IndexReader.indexExists method(s) and see if you still
> get the error.  If you don't, please put the patch with your changes in a
> new JIRA issues, so we can fix the problem.
>
> Thanks,
> Otis
>
>
> ----- Original Message ----
> From: wenjie zheng <wenjie.zheng@gmail.com>
> To: java-dev@lucene.apache.org
> Sent: Tuesday, May 9, 2006 10:57:05 AM
> Subject: Re: weird behavior of IndexReader.indexExists()
>
> I am a little bit confused here.
> Since I didn't change the lock directory or disable locking and it is on
> the
> single server sharing same fs, does it mean I am not supposed to get the
> error?
> But I did get the error. What should I do?
>
> Wenjie
> On 5/9/06, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
> >
> > If you don't explicitly change the lock directory and do not disable
> > locking, the same directory should be used.  I'm assuming this is all
> done
> > on a single server sharing the same file system.  The locks are stored
> in
> > the system's default temporary directory.  That's typically /tmp under
> > UNIX/OSX and C:\WINDOWS\TEMP (I think) on Winblows.
> >
> > Otis
> >
> > ----- Original Message ----
> > From: wenjie zheng <wenjie.zheng@gmail.com>
> > To: java-dev@lucene.apache.org
> > Sent: Tuesday, May 9, 2006 9:09:47 AM
> > Subject: Re: weird behavior of IndexReader.indexExists()
> >
> > Thanks,
> > I guess my question is how to make sure both instances are using the
> same
> > lock directory.
> >
> > Wenjie
> >
> > On 5/9/06, Vanlerberghe, Luc <Luc.Vanlerberghe@bvdep.com> wrote:
> > >
> > > Make sure both instances are using the same lock directory.
> > > The segments file should only be read or written while holding the
> > > commit lock.
> > >
> > > If the lock directories don't match, you'll get more 'strange'
> errors...
> > >
> > > In Lucene 1.4.2 some methods did not use the lock, this has been
> patched
> > > a couple of months ago.
> > >
> > > Luc
> > >
> > > -----Original Message-----
> > > From: Andy Hind [mailto:andy.hind@alfresco.org]
> > > Sent: dinsdag 9 mei 2006 13:47
> > > To: java-dev@lucene.apache.org
> > > Subject: RE: weird behavior of IndexReader.indexExists()
> > >
> > > Hi
> > >
> > > I think I have discovered this too.
> > > It is on my list of issues to raise ....
> > >
> > > The index exist test looks for the segment file.
> > > When the index is committing, and you are unlucky, this file may not
> be
> > > found as the new segments file replaces the old one. The result is the
> > > index appears not to exist.
> > >
> > > Regards
> > >
> > > Andy
> > >
> > > -----Original Message-----
> > > From: wenjie zheng [mailto:wenjie.zheng@gmail.com]
> > > Sent: 08 May 2006 18:57
> > > To: java-dev@lucene.apache.org
> > > Subject: Re: weird behavior of IndexReader.indexExists()
> > >
> > > This happens sometimes when number of docs is over 2000. So it's kinda
> > > of
> > > random.
> > >
> > > Wenjie
> > >
> > > On 5/8/06, wenjie zheng <wenjie.zheng@gmail.com> wrote:
> > > >
> > > > I created an index with more than 30,000 text files.
> > > > I used indexExists() to determine either to create a new index or to
> > > add
> > > > docs to the existing index.
> > > >
> > > > But when the num of docs in the index was over 3,000 (sometimes
> 3,400,
> > > > sometimes 3,200), the indexExists function returns false, so I ended
> > > up
> > > > recreating a new index.
> > > >
> > > > Here is my code:
> > > > if index exists, we will add files to it, otherwise, create a new
> > > index.
> > > > In either case, an IndexingThread will be spawn to do that.
> > > >          if(IndexReader.indexExists(indexDir)){
> > > >                 logger.info("Working on existing index ...");
> > > >                 IndexingThread.startIndexingThread(Username, new
> > > > File(propsFile), new File(indexDir), docs,
> > > >                               new StandardAnalyzer(), false);
> > > >           }else{
> > > >                 logger.info("Create a new index ...");
> > > >                 IndexingThread.startIndexingThread(Username, new
> > > > File(propsFile), new File(indexDir), docs,
> > > >                                new StandardAnalyzer(), true);
> > > >            }
> > > >
> > > >
> > > > inside the startIndexingThread function, I am calling the following
> > > > function to add files to the index:
> > > >     /**
> > > >      * Add an array of Files to an index
> > > >      *
> > > >      * @param propsFile the properties file
> > > >      * @param indexDir  the folder where index files will be created
> > > in
> > > >      * @param docs      an array of Files to be add to the index
> > > >      * @param analyzer  any Analyzer object
> > > >      */
> > > >     public void addFiles(File propsFile, File indexDir, File[] docs,
> > > > Analyzer analyzer, boolean overwrite) throws Exception {
> > > >         Properties props = new Properties(new
> > > FileInputStream(propsFile));
> > > >
> > > >         if(overwrite || IndexReader.indexExists(indexDir)){ //either
> > > > overwrite or working on an existing index
> > > >             Directory index = FSDirectory.getDirectory(indexDir,
> > > > overwrite);
> > > >             IndexWriter writer = new IndexWriter(index, analyzer,
> > > > overwrite);
> > > >
> > > >             FileIndexer indexer = new FileIndexer(props);
> > > >
> > > >             long start = new Date().getTime();
> > > >             indexer.index(writer, docs);
> > > >             writer.optimize();
> > > >             writer.close ();     // close the writer
> > > >             index.close();      // close the index Directory
> > > >             long end = new Date().getTime();
> > > >
> > > >             logger.info("Total time: " + (end - start) + " ms");
> > > >
> > > >         }else{
> > > >             logger.error("Index files are not found: " +
> > > > indexDir.getAbsolutePath() + ", overwrite = false");
> > > >         }
> > > >     }
> > > >
> > > > Thanks,
> > > > Wenjie
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> > >
> > >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message