lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Elschot <paul.elsc...@xs4all.nl>
Subject Re: Lucene on Linux problem...
Date Sat, 02 Apr 2005 09:02:54 GMT
On Saturday 02 April 2005 10:29, Kristian Ottosen wrote:
> Hello,
> 
> I have really been pulling out hair for a while over this. The problem only
> occur using FSDirectory on some Linux system (some Debian, Suse and Redhat -
> but not all) and never under Windows, Solaris or Mac OS X. 
> 
> What happens is that Lucene for some reason tries to read a segment which
> was just merged into another segment and therefore deleted and fails with a
> FileNotFoundException.
>  
> A few facts: Only a single thread is involved. If we use a RAM based Store
> (RAMDirectory) there is no problem at all. Also there seems to a problem
> related to locking using the lock files. We get timeouts even though there
> is only a single thread working on the index. Switching the compound file
> format on/off makes no difference. The problem may not appear immediately -
> but only after several thousand documents have been indexed in a row using a
> combination of IndexSearcher (check for duplicates) and IndexWriter (add
> document) operations.
> 
> I wonder if there is general problem running Lucene on top of some of the
> journaling file systems like ReiserFS or Ext3?

The only reason I can think of for a FileNotFoundException is that the 
corresponding close() of the file has not terminated yet, or was not done
at all.

I don't know the precise file system semantics in relation to a single Linux 
process, but if these semantics allow for a file closed by a process to be
(temporarily) invisible to that same process, one could end up with
a FileNotFoundException.

It's the first time AFAIK something like this has been reported, so another
source of the problem could be your hard disk, but I would not expect a
journalling file system to ignore errors like this, ie. a journalling file
system should provide some indication of a error situation for a
hardware problem. Could you might check the system log for
file system messages?

Going back to the file system semantics allowing a temporarily invisible
file, you might change the Lucene java code to perform a file system sync
(via a shell) in case of the FileNotFoundException, and then retry opening
the file?
If that helps, it's the file system semantics. In case there is such a thing
as a file system sync in the java.io package, this would have to be added
to Lucene. Otherwise it would probably be useful to inform the maintainers
of the JVM.

Regards,
Paul Elschot.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message