lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Ritchie <>
Subject Re: Compound file index issue
Date Thu, 15 Jan 2004 06:21:16 GMT
Dmitry Serebrennikov wrote:

> I've seen a couple of other people reporting similar issues when 
> switching to the latest 1.3 tree. However, usually the problems come 
> from lack of synchronization of multiple threads doing reads and writes 
> against the index. But it's definetely possible that the compound index 
> format makes this issue more visible or that it has its own bugs. In my 
> testing and in the initial testing of the Lucene dev team when we were 
> working on the compound index format we didn't see any such issues, but 
> of course that could just mean that we didn't look in the right place.

While I'm fairly certain our app fulfills all the locking requirements for lucene, it's possible

that we've missed something as well. I'll go through the code again to see if there is any
place I 
allow multiple threads access improperly.

> Anyway, it would be extremely helpful if you could produce a 
> self-contained program that demonstrates this behavior. I know that this 
> can be hard to impossible with concurrency-related issues, but still.

I'll see what I can do.

> Also, what OS is this running under? This issue is related to filesystem 
> locking behavior which is different between Unix and Windows.

Red Hat Linux release 9 (Shrike) under their 2.4.20-24.9smp kernel and Sun's 1.4.2_03 SDK.
We did 
disable lucene's file based locking mechanism since our app is not setup to allow multiple
JVM's to write to a single index (one index per JVM) and we've had no end of issues with the
based locking. This was done by just setting FSDirectory.DISABLE_LOCKS to true;

> Does this happen every time or does it happen only under heavy load, etc.?
> What is the system doing when this occurs (i.e. indexing and searching 
> at the same time or just searching etc)?

Unfortunately, I have no answers for any of these questions. The best I can say is that the
load on 
the server is usually very low (load average of 0.03) and the box itself has fast scsi based
(IO is much faster for instance than my local development machine). The only other pertinent
of info that I can think of is that this instance of the application was setup to update the
with new and updated content once every 10 minutes.


Bruce Ritchie

> Bruce Ritchie wrote:
>> We've started using the new compound indexing functionality available 
>> in lucene 1.3 final however we've been seeing an issue that I haven't 
>> quite resolved and was hoping that people on this list might be able 
>> to lend help on.
>> The stack trace of the issue is:
>> 2004.01.13 18:15 
>> []
>> /var/search/BfUfO9s1/_qb5.fnm (No such 
>> file or directory)
>>         at Method)
>>         at<init>(
>>         at 

>>         at 
>>         at 
>>         at org.apache.lucene.index.FieldInfos.<init>(
>>         at 
>> org.apache.lucene.index.SegmentReader.<init>(
>>         at 
>> org.apache.lucene.index.SegmentReader.<init>(
>>         at 
>> org.apache.lucene.index.IndexReader$1.doBody(
>>         at$
>>         at
>>         at 

>>         ...
>> The error seems obvious enough from a quick glance of the above code - 
>> the file was been deleted. However, since we are using compound 
>> indexing there should be a /var/search/BfUfO9s1/_qb5.cfs file, however 
>> that file does not exist. The only files in the search directory are:
>> -bash-2.05b$ ls -l
>> total 6908
>> -rw-r--r--    1 resin    js              4 Jan 13 18:14 deletable
>> -rw-r--r--    1 resin    js        7029055 Jan 13 10:03 _qau.cfs
>> -rw-r--r--    1 resin    js              9 Jan 13 18:14 _qb5.del
>> -rw-r--r--    1 resin    js          16060 Jan 13 18:14 _qb7.cfs
>> -rw-r--r--    1 resin    js             34 Jan 13 18:14 segments
>> From what I can tell it seems to me that the segments are somehow 
>> getting out of sync with what is actually on the filesystem somehow.
>> Any suggestions are where to start looking for the cause of this bug?
>> Regards,
>> Bruce Ritchie
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

AOL - bruceritchie101
ICQ - 9929791

View raw message