lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Lucene 2.3.1 Index Corruption?
Date Tue, 18 Mar 2008 11:52:19 GMT

It looks like you ignore any IOException coming out of  
IndexWriter.close?  Can you put some code in the catch clause around  
writer.close to see if you are hitting some exception there?

Also, you forcefully remove the write lock if it's present.  But are  
you absolutely certain there isn't another writer actually writing to  
that index directory?

Do you copy the index or alter it in some way?  One strange thing in  
your directory listing was the file "indexinfo", which isn't a Lucene  
index file.  Something else must be writing that file.

Mike

Jamie wrote:

> Hi Michael
>
> Sorry for the late reply. As you guessed, it missed my attention.
>
> Michael McCandless wrote:
>>
>> Hi,
>>
>> Can you describe what led up to this?
>
> My application indexes emails. In this particular instance, I had  
> reindexed all emails from their original sources. The error  
> occurred while I was using a search to search through the index.
>> Were there any exceptions when adding documents to the index?
> I had a look through all my application debug logs and there were  
> no exceptions outputted.
>
>>   Was the index newly created with 2.3.1 or created on 2.3.0 or 2.2?
> This index was created by v2.3.1
>>
>> What options are you using in your IndexWriter?
>  See source code below:
>
>      public void indexMessage(Email email) throws  
> MessageSearchException {
>          Volume volume = email.getEmailId().getVolume();
>          VolumeIndex volumeIndex = volumeIndexes.get(volume);
>          if (volumeIndex!=null) {
>              volumeIndex.indexMessage(email);
>          } else {
>              volumeIndex = new VolumeIndex(volume);
>              volumeIndex.indexMessage(email);
>              volumeIndexes.put(volume,volumeIndex);
>          }
>      }
>          public class VolumeIndex {
>                        IndexWriter writer;
>                Volume volume;
>                Timer closeIndexTimer = new Timer();
>             AccessStatus volumeOpened = AccessStatus.CLOSED;
>             Object indexLock = new Object();
>                          public synchronized AccessStatus  
> getAccessStatus() { return volumeOpened;}
>
>              public synchronized void setAccessStatus(AccessStatus  
> volumeOpened) {
>                  this.volumeOpened = volumeOpened;
>             }
>                            public VolumeIndex(Volume volume) {
>                        this.volume = volume;
>                      closeIndexTimer.scheduleAtFixedRate(new  
> TimerTask() {
>                        public void run() {
>                             closeIndex(writer);
>                        }
>                    }, indexOpenTime, indexOpenTime);
>                        }
>
>              protected void openIndex() throws  
> MessageSearchException {
>                  synchronized(indexLock) {
>                        if (getAccessStatus()==AccessStatus.CLOSED) {
>                            logger.debug("openIndex() index will be  
> opened. it is currently closed.");
>                            openIndex(false);
>                            setAccessStatus(AccessStatus.OPEN);
>                        } else
>                            logger.debug("openIndex() did not bother  
> opening index. it is already open.");
>                  }
>                              }
>                          protected void openIndex(boolean retry)  
> throws MessageSearchException {
>                    if (volume == null)
>                        throw new MessageSearchException("assertion  
> failure: null volume",logger);
>                    logger.debug("opening index for write {"+volume 
> +"}");
>                    prepareIndex(volume);
>                    Index activeIndex = volume.getActiveIndex();
>                    logger.debug("opening search index for write  
> {indexpath='"+activeIndex.getPath()+"'}");
>                    try {
>                            writer = new IndexWriter 
> (activeIndex.getPath(), analyzer);
>                    } catch (IOException io)
>                    {
>                        if (!retry) {
>                            // most obvious reason for error is that  
> there is a lock on the index, due hard shutdown
>                            // resolution delete the lock, and try  
> again
>                            logger.warn("failed to open search index  
> for write. possible write lock due to hard system shutdown.",io);
>                            logger.info("attempting recovery.  
> deleting index lock file and retrying..");
>                            File lockFile = new File 
> (activeIndex.getPath()+File.separatorChar + "write.lock");
>                            lockFile.delete();
>                            try {
>                                openIndex(true);
>                            } catch (MessageSearchException mse) {
>                                throw mse;
>                            }
>                        }
>                        throw new MessageSearchException("failed to  
> open/ index writer {location='"+activeIndex.getPath()+"'}",io,logger);
>                    }
>            }
>
>              public void prepareIndex(Volume volume) throws  
> MessageSearchException {
>                                if (volume==null)
>                            throw new MessageSearchException 
> ("assertion failure: null volume",logger);
>                                if (volume.getIndexPath().startsWith 
> ("rmi://"))
>                          return;
>                                          File indexDir = new File 
> (volume.getIndexPath());
>                  if (!indexDir.exists()) {
>                    logger.info("index directory does not exist.  
> will proceed with creation {location='" + volume.getIndexPath() +  
> "'}");
>                    boolean success = indexDir.mkdir();
>                    if (!success)
>                            throw new MessageSearchException("failed  
> to create index directory {location='" + volume.getIndexPath() +  
> "'}",logger);
>                    logger.info("index directory successfully  
> created {location='" + volume.getIndexPath() + "'}");
>                  }
>                        }
>                        public void indexMessage(Email message)  
> throws MessageSearchException  {
>                long s = (new Date()).getTime();
>                if (message == null)
>                    throw new MessageSearchException("assertion  
> failure: null message",logger);
>                logger.debug("indexing message {"+message+"}");
>                              Document doc = new Document();
>                try {
>                   writeMessageToDocument 
> (message,doc);                   String language = doc.get("lang");
>                   if (language==null)
>                       language = getIndexLanguage();
>                           synchronized (indexLock) {
>                               openIndex();
>                               writer.addDocument 
> (doc,AnalyzerFactory.getAnalyzer 
> (language,AnalyzerFactory.Operation.INDEX));
>                           }
>                   logger.debug("message indexed successfully  
> {"+message+",language='"+language+"'}");
>                } catch (MessagingException me)
>                {
>                   throw new MessageSearchException("failed to  
> decode message during indexing",me,logger);
>                } catch (IOException me) {
>                    throw new MessageSearchException("failed to  
> index message {"+message+"}",me,logger);
>                } catch (ExtractionException ee)
>                {
>                   throw new MessageSearchException("failed to  
> decode attachments in message {"+message+"}",ee,logger);
>                } catch (Exception e) {
>                    throw new MessageSearchException("failed to  
> index message",e,logger);
>                }
>                logger.debug("indexing message end {"+message+"}");
>                              long e = (new Date()).getTime();
>                logger.debug("indexing time {time='"+(e-s)+"'}");
>            }
>                          protected void closeIndex(IndexWriter  
> writer) {
>
>                       synchronized(indexLock) {
>                                                 if (getAccessStatus 
> ()==AccessStatus.CLOSED)
>                                return;
>                                              try {
>                                if (writer!=null)
>                                    writer.close();
>                                try { Thread.sleep(50); } catch  
> (Exception e) {}
>                        } catch (Exception io) {}
>                        setAccessStatus(AccessStatus.CLOSED);
>                       }
>               }
>                      protected void finalize() throws Throwable {
>                logger.debug("volumeindex class is shutting down");
>                try {
>                    closeIndexTimer.cancel();
>                } finally {
>                super.finalize();
>                }
>            }
>                }
>
>>
>> Is it easy to reproduce?
> Its difficult to reproduce since the problem seems intermittant..
>> If so, can you call setInfoStream on your IndexWriter when  
>> creating this index and post the resulting output?
> I'll try this but I cannot guarantee anything. Do you see anything  
> obvious from the above?
>>
>> Mike
>>
>> Jamie wrote:
>>
>>>
>>> Hi There
>>>
>>> I am getting the following error while searching a given index:
>>>
>>> java.io.FileNotFoundException: /usr/local/index/_0.fdt (No such  
>>> file or directory)
>>>        at java.io.RandomAccessFile.open(Native Method)
>>>        at java.io.RandomAccessFile.<init>(Unknown Source)
>>>        at org.apache.lucene.store.FSDirectory$FSIndexInput 
>>> $Descriptor.<init>(FSDirectory.java:506)
>>>        at org.apache.lucene.store.FSDirectory$FSIndexInput.<init> 
>>> (FSDirectory.java:536)
>>>        at org.apache.lucene.store.FSDirectory.openInput 
>>> (FSDirectory.java:445)
>>>        at org.apache.lucene.index.FieldsReader.<init> 
>>> (FieldsReader.java:75)
>>>        at org.apache.lucene.index.SegmentReader.initialize 
>>> (SegmentReader.java:308)
>>>        at org.apache.lucene.index.SegmentReader.get 
>>> (SegmentReader.java:262)
>>>        at org.apache.lucene.index.SegmentReader.get 
>>> (SegmentReader.java:197)
>>>        at org.apache.lucene.index.MultiSegmentReader.<init> 
>>> (MultiSegmentReader.java:55)
>>>        at org.apache.lucene.index.DirectoryIndexReader$1.doBody 
>>> (DirectoryIndexReader.java:75)
>>>        at org.apache.lucene.index.SegmentInfos 
>>> $FindSegmentsFile.run(SegmentInfos.java:636)
>>>        at org.apache.lucene.index.DirectoryIndexReader.open 
>>> (DirectoryIndexReader.java:63)
>>>        at org.apache.lucene.index.IndexReader.open 
>>> (IndexReader.java:209)
>>>        at org.apache.lucene.index.IndexReader.open 
>>> (IndexReader.java:173)
>>>        at org.apache.lucene.search.IndexSearcher.<init> 
>>> (IndexSearcher.java:48)
>>>
>>> My software used to work perfectly under earlier versions of  
>>> Lucene. Since I upgraded to 2.3.1, this problem has arisen.
>>>
>>> I seriously worried my customer's indexes will be corrupted.  
>>> Lucene expects to find a file that does not exist.
>>>
>>> Any ideas on what might be happening and how to rectify this?
>>>
>>> Jamie
>>>
>>>
>>> -------------------------------------------------------------------- 
>>> -
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> -- 
> Stimulus Software - MailArchiva
> Email Archiving And Compliance
> USA Tel: +1-713-366-8072 ext 3
> UK Tel: +44-20-80991035 ext 3
> Email: jamie@stimulussoft.com
> Web: http://www.mailarchiva.com
>
> To receive MailArchiva Enterprise Edition product announcements,  
> send a message to: <mailarchiva-enterprise-edition- 
> subscribe@stimulussoft.com>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message