lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Serious Index Corruption Error - FileNotFoundException
Date Wed, 07 May 2008 11:56:43 GMT

On quickly looking through the code I think there are some serious  
hazards that could lead to this exception.

First, in your openIndex code, if you hit a LockObtainFailedException  
in trying to open your writer, you are forcefully removing the write  
lock and then retrying.  Yet, you also open an IndexReader to delete  
documents, which acquires the write lock.  If ever you have this  
IndexReader open, and then you forcefully remove the write lock and  
open the writer, that would cause this exception.

Second, you have a deletIndex method, which first tries to use the  
writer with create=true (good) but then falls back to manually  
removing the files.  Why is that fallback necessary?  If, for  
example, you are also hitting a LockObtainFailedException, then  
forcefully removing files while an IndexReader or IndexWriter holds  
the write lock is also dangerous and would lead to this exception.

In general it's very dangerous to forcibly remove, or ignore,  
Lucene's write lock.  It really should only be necessary when  
something catastrophic occurred (JVM crashed).

Also, note that IndexWriter can now delete documents.  This would  
simplify your code and possibly fix these two hazards.

Do you see any of the error/warnings that you send to your logger?   
(They would be corroborating evidence here).

Mike

Jamie wrote:
> Hi there
>
> It appears my Lucene 2.3.1 index is corrupted. I get the following  
> error when searching:
>
> /mnt/indexnew/_3wk0.cfs (No such file or directory)
> java.io.FileNotFoundException: /mnt/indexnew/_3wk0.cfs (No such  
> file or directory)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
>        at org.apache.lucene.store.FSDirectory$FSIndexInput 
> $Descriptor.<init>(FSDirectory.java:506)
>        at org.apache.lucene.store.FSDirectory$FSIndexInput.<init> 
> (FSDirectory.java:536)
>        at org.apache.lucene.store.FSDirectory.openInput 
> (FSDirectory.java:445)
>        at org.apache.lucene.index.CompoundFileReader.<init> 
> (CompoundFileReader.java:70)
>        at org.apache.lucene.index.SegmentReader.initialize 
> (SegmentReader.java:277)
>        at org.apache.lucene.index.SegmentReader.get 
> (SegmentReader.java:262)
>        at org.apache.lucene.index.SegmentReader.get 
> (SegmentReader.java:197)
>        at org.apache.lucene.index.MultiSegmentReader.<init> 
> (MultiSegmentReader.java:55)
>        at org.apache.lucene.index.DirectoryIndexReader$1.doBody 
> (DirectoryIndexReader.java:75)
>        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run 
> (SegmentInfos.java:636)
>        at org.apache.lucene.index.DirectoryIndexReader.open 
> (DirectoryIndexReader.java:63)
>        at org.apache.lucene.index.IndexReader.open(IndexReader.java: 
> 209)
>        at org.apache.lucene.index.IndexReader.open(IndexReader.java: 
> 173)
>        at org.apache.lucene.search.IndexSearcher.<init> 
> (IndexSearcher.java:48)
>        at
>
> Question #1: How can I recover from this error without regenerating  
> the entire index?
>
> Is there a utility I could use?
>
> Question #2: What is the reason why I am missing a segument cfs file?
>
> Could it be a problem with the synchronization of threads during  
> Indexing (see attached code)?
> Could it be a bug in MailArchiva?
> I have tried to implement locks in appropriate places.
> java.io.FileNotFoundException: /mnt/indexnew/_3wk0.cfs (No such  
> file or directory)
>
> Any help would be much appreciated!
>
> Jamie
>
> package com.stimulus.archiva.index;
>
> /* Copyright (C) 2005-2007 Jamie Angus Band
>  * MailArchiva Open Source Edition Copyright (c) 2005-2007 Jamie  
> Angus Band
>  * This program is free software; you can redistribute it and/or  
> modify it under the terms of
>  * the GNU General Public License as published by the Free Software  
> Foundation; either version
>  * 2 of the License, or (at your option) any later version.
>  *
>  * This program is distributed in the hope that it will be useful,  
> but WITHOUT ANY WARRANTY;
>  * without even the implied warranty of MERCHANTABILITY or FITNESS  
> FOR A PARTICULAR PURPOSE.
>  * See the GNU General Public License for more details.
>  *
>  * You should have received a copy of the GNU General Public  
> License along with this program;
>  * if not, see http://www.gnu.org/licenses or write to the Free  
> Software Foundation,Inc., 51
>  * Franklin Street, Fifth Floor, Boston, MA 02110-1301  USA
>  */
>
>
> import javax.mail.*;
> import javax.mail.internet.*;
> import java.io.*;
> import java.util.*;
> import java.util.zip.*;
> import com.ice.tar.*;
> import org.apache.log4j.Logger;
> import org.apache.lucene.document.*;
> import org.apache.lucene.index.*;
> import org.apache.lucene.store.*;
> import com.stimulus.archiva.domain.*;
> import com.stimulus.archiva.exception.*;
> import com.stimulus.archiva.language.*;
> import com.stimulus.archiva.extraction.*;
> import com.stimulus.archiva.search.*;
> import com.stimulus.util.TempFiles;
> import com.stimulus.archiva.domain.fields.*;
> import com.stimulus.util.*;
> import java.nio.charset.Charset;
> import org.apache.log4j.Level;
>
> public class MessageIndex extends Indexer implements Serializable {
> 	
> 	
> 	  private static final long serialVersionUID = -17692874371162272L;
> 	  protected static final Logger logger = Logger.getLogger 
> (MessageIndex.class.getName());
> 	  protected static int INDEX_WAIT_PERIOD = 50;
> 	  protected static int DEAD_PERIOD = 300000000;
> 	  protected boolean shutdown = false;
> 	  static Hashtable<Volume,VolumeIndex> volumeIndexes = new  
> Hashtable<Volume,VolumeIndex>();
> 	  static Object volumeIndexLock = new Object();
>
>     public MessageIndex() {
> 	
>     }
>
>
> 	public void deleteIndex(Volume volume) throws  
> MessageSearchException {
> 		  VolumeIndex volumeIndex = getVolumeIndex(volume);
> 		  volumeIndex.deleteIndex();
> 	}
> 		
>  	
> 	public VolumeIndex getVolumeIndex(Volume volume) {
> 		 VolumeIndex volumeIndex = null;
> 		  synchronized (volumeIndexLock) {
> 			  volumeIndex = volumeIndexes.get(volume);
> 			  if (volumeIndex==null) {
> 				  volumeIndex = new VolumeIndex(this,volume);
> 				  volumeIndexes.put(volume,volumeIndex);
> 			  }
> 		  }
> 		  return volumeIndex;
> 	}
> 	  	
> 	public void indexMessage(Email email) throws MessageSearchException {
> 		  VolumeIndex volumeIndex = getVolumeIndex(email.getEmailId 
> ().getVolume());
> 		  volumeIndex.indexMessage(email);
> 	}
> 		
> 		
> 		  // Enterprise version
> 	
> 	public void deleteMessage(EmailID emailID) throws  
> MessageSearchException {
> 		  if (emailID == null)
> 	            throw new MessageSearchException("assertion failure:  
> null emailID",logger);
> 	  logger.debug("delete message {'"+emailID+"'}");
> 	  Volume volume = emailID.getVolume();
> 	  File indexDir = new File(volume.getIndexPath());
> 	  if (!indexDir.exists())
> 		  throw new MessageSearchException("could not delete email from  
> index. volume does not exist. {'"+emailID+"}",logger);
> 	  IndexReader indexReader = null;
> 	  try {
> 		  indexReader = IndexReader.open(indexDir);
> 	  } catch (IOException e ) {
> 		  throw new MessageSearchException("failed to open index to  
> delete email",e,logger);
> 	  }
> 	  try {
> 		  indexReader.deleteDocuments(new Term("uid",emailID.getUniqueID 
> ()));
> 		  indexReader.close();
> 	  } catch (Exception e) {
> 		  throw new MessageSearchException("failed to delete email from  
> index.",e,logger);
> 	  }
> 	}
> 	
> 	public void prepareIndex(Volume volume) throws  
> MessageSearchException {
> 			
> 		  if (volume==null)
> 		            throw new MessageSearchException("assertion failure:  
> null volume",logger);
> 	
> 	  if (volume.getIndexPath().startsWith("rmi://"))
> 			  return;
> 			
> 	  File indexDir = new File(volume.getIndexPath());
> 	  if (!indexDir.exists()) {
> 		logger.info("index directory does not exist. will proceed with  
> creation {location='" + volume.getIndexPath() + "'}");
> 		boolean success = indexDir.mkdir();
> 		if (!success)
> 				throw new MessageSearchException("failed to create index  
> directory {location='" + volume.getIndexPath() + "'}",logger);
> 		logger.info("index directory successfully created {location='" +  
> volume.getIndexPath() + "'}");
> 		  }
> 	
> 	}
> 	
> 	protected void finalize() throws Throwable {
> 	    logger.debug("messagindex class is shutting down");
> 	    try {
> 	    	shutdown = true;
> 	    } finally {
> 		super.finalize();
> 	    }
> 	}
>
> }
> package com.stimulus.archiva.index;
>
> import java.io.File;
> import java.io.IOException;
> import java.io.PrintStream;
> import javax.mail.MessagingException;
> import org.apache.log4j.Level;
> import org.apache.log4j.Logger;
> import org.apache.lucene.document.Document;
> import org.apache.lucene.index.*;
> import org.apache.lucene.store.FSDirectory;
> import com.stimulus.archiva.domain.Config;
> import com.stimulus.archiva.domain.Email;
> import com.stimulus.archiva.domain.Indexer;
> import com.stimulus.archiva.domain.Volume;
> import com.stimulus.archiva.exception.ExtractionException;
> import com.stimulus.archiva.exception.MessageSearchException;
> import com.stimulus.archiva.language.AnalyzerFactory;
> import com.stimulus.archiva.search.*;
> import java.util.*;
> import org.apache.lucene.store.AlreadyClosedException;
>
> public class VolumeIndex {
> 	
> 		 protected static final Logger logger = Logger.getLogger 
> (VolumeIndex.class.getName());
> 		 public static final int indexOpenTime = 2000;
> 	   	 IndexWriter writer = null;
> 	   	 Volume volume;
> 	   	 Timer closeIndexTimer = new Timer();
> 		 Object indexLock = new Object();
> 		
> 		
> 		 ArchivaAnalyzer analyzer 	= new ArchivaAnalyzer();
> 		 Indexer indexer = null;
> 	  	  public VolumeIndex(Indexer indexer, Volume volume) {
> 	  		  	this.volume = volume;
> 	  		  	this.indexer = indexer;
> 		  		closeIndexTimer.scheduleAtFixedRate(new TimerTask() {
> 		            public void run() {
> 		             	closeIndex();
> 		            }
> 		        }, indexOpenTime, indexOpenTime);
> 	  	  }
>
> 		  protected void openIndex(boolean retry) throws  
> MessageSearchException {
> 			 Exception lastError = null;
> 			 synchronized(indexLock) {
> 		    	if (writer==null) {
> 		    		logger.debug("openIndex() index will be opened. it is  
> currently closed.");
> 		    	} else {
> 		    		logger.debug("openIndex() did not bother opening index. it  
> is already open.");
> 		    		return;
> 		    	}
>
> 				logger.debug("opening index for write {"+volume+"}");
> 				indexer.prepareIndex(volume);
> 				logger.debug("opening search index for write  
> {indexpath='"+volume.getIndexPath()+"'}");
> 			
> 				try {
> 						writer = new IndexWriter(FSDirectory.getDirectory 
> (volume.getIndexPath()),false, analyzer);
> 						writer.setMaxFieldLength(50000);
> 						if (logger.isDebugEnabled()) {
> 							File file = new File(Config.getFileSystem().getLogPath() 
> +File.separator+"index.log");
> 							PrintStream debugout = new PrintStream(file);
> 							writer.setInfoStream(debugout);
> 						}
> 				} catch (IOException io) {
> 					lastError = io;
> 					if (!retry) {
> 						// most obvious reason for error is that there is a lock on  
> the index, due hard shutdown
> 						    // resolution delete the lock, and try again
> 						    logger.warn("failed to open search index for write.  
> possible write lock due to hard system shutdown.",io);
> 						    logger.info("attempting recovery. deleting index lock  
> file and retrying..");
> 						    File lockFile = new File(volume.getIndexPath() 
> +File.separatorChar + "write.lock");
> 						    lockFile.delete();
> 						    openIndex(true);
> 					} else {
> 						throw new MessageSearchException("failed to open/ index  
> writer {location='"+volume.getIndexPath()+"'}",lastError,logger);
> 					}
> 				}
> 			}
> 		}
> 		
> 		public void indexMessage(Email message) throws  
> MessageSearchException  {
>
> 			long s = (new Date()).getTime();
> 			if (message == null)
> 			    throw new MessageSearchException("assertion failure: null  
> message",logger);
> 			logger.debug("indexing message {"+message+"}");
> 			
> 			Document doc = new Document();
> 			try {
> 			
> 			   DocumentIndex docIndex = new DocumentIndex(indexer);
> 			   docIndex.write(message,doc);
> 			   String language = doc.get("lang");
> 			   if (language==null)
> 				   language = indexer.getIndexLanguage();
> 		   		synchronized (indexLock) {
> 		   			openIndex(false);
> 		   			writer.addDocument(doc,AnalyzerFactory.getAnalyzer 
> (language,AnalyzerFactory.Operation.INDEX));
> 		   		}
> 		   		doc = null;
> 			   logger.debug("message indexed successfully {"+message 
> +",language='"+language+"'}");
> 			} catch (MessagingException me)
> 			{
> 			   throw new MessageSearchException("failed to decode message  
> during indexing",me,logger, Level.DEBUG);
> 			} catch (IOException me) {
> 			    throw new MessageSearchException("failed to index message  
> {"+message+"}",me,logger, Level.DEBUG);
> 			} catch (ExtractionException ee)
> 			{
> 				// we will want to continue indexing
> 			   //throw new MessageSearchException("failed to decode  
> attachments in message {"+message+"}",ee,logger, Level.DEBUG);
> 			} catch (AlreadyClosedException ace) {
> 				indexMessage(message);
> 			} catch (Exception e) {
> 			    throw new MessageSearchException("failed to index  
> message",e,logger, Level.DEBUG);
> 			}
> 			logger.debug("indexing message end {"+message+"}");
> 			
> 			long e = (new Date()).getTime();
> 		    logger.debug("indexing time {time='"+(e-s)+"'}");
> 		}
> 			
> 		protected void closeIndex() {
> 		
> 			   synchronized(indexLock) {
> 			        if (writer==null)
> 			        		return;
> 			        try {
> 			        	writer.close();
> 					    logger.debug("writer closed");
> 					} catch (Exception io) {
> 						logger.error("failed to close index writer:"+io.getMessage 
> (),io);
> 	
> 					}
> 			        writer = null;
> 			   	}
> 	   	}
> 		 //	 deliberately non recursive (so we avoid situations where the  
> whole h/d is deleted)
> 		  public void deleteIndex() throws MessageSearchException {
> 			  logger.debug("delete index {indexpath='"+volume.getIndexPath() 
> +"'}");
> 			
> 				try {
> 					  writer = new IndexWriter(FSDirectory.getDirectory 
> (volume.getIndexPath()),false, analyzer,true);
> 				} catch (Exception cie) {
> 					 logger.error("failed to delete index  
> {index='"+volume.getIndexPath()+"'}",cie);
> 					 File indexDir = new File(volume.getIndexPath());
>              		 if (!indexDir.exists()) return;
>              		 if (indexDir.isDirectory()) {
>           	            String[] children = indexDir.list();
>           	            for (int i=0; i<children.length; i++) {
>           	                String filepath = volume.getIndexPath() 
> +File.separatorChar+children[i];
>           	                logger.debug("deleting file {path='" +  
> filepath +"'}");
>           	                File file = new File(filepath);
>           	                boolean success = file.delete();
>           	                if (!success) {
>           	                	try {
>           	                		File newFile = File.createTempFile 
> ("temp","idx");
>           	                		file.renameTo(newFile);
>           	                	} catch (Exception e2) {
>           	                		throw new MessageSearchException 
> ("failed to delete file in existing index {filepath='"+filepath 
> +"'}",logger);
>           	                	}
>           	                } else
>                                    logger.debug("deleted file  
> successfully {filepath='" + filepath +"'}");
>           	            }
>              		 }
>              		 return;
> 				}
> 				try {
> 				  writer.close();
>               	} catch (Exception e) {
>               		logger.error("failed to delete index  
> {index='"+volume.getIndexPath()+"'}",e);
>               	}
>
>               	MessageIndex.volumeIndexes.remove(this);
>
>
> 		  }
> 		
> 		protected void finalize() throws Throwable {
> 		    logger.debug("volumeindex class is shutting down");
> 		    try {
> 		    	closeIndexTimer.cancel();
> 		    	closeIndex();
> 		    } finally {
> 			super.finalize();
> 		    }
> 		}
> 		
> }
>
> 	
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message