Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Message-ID: <4006315C.6070408@jivesoftware.com>
Date: Thu, 15 Jan 2004 01:21:16 -0500
From: Bruce Ritchie <bruce@jivesoftware.com>
Organization: Jive Software
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
 rv:1.5) Gecko/20031007
MIME-Version: 1.0
To: Lucene Developers List <lucene-dev@jakarta.apache.org>
Subject: Re: Compound file index issue
References: <40056471.5020704@jivesoftware.com>
 <4005D862.2000404@earthlink.net>
In-Reply-To: <4005D862.2000404@earthlink.net>
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
 micalg=sha1; boundary="------------ms090506090501080809090607"

--------------ms090506090501080809090607
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Dmitry Serebrennikov wrote:

> I've seen a couple of other people reporting similar issues when 
> switching to the latest 1.3 tree. However, usually the problems come 
> from lack of synchronization of multiple threads doing reads and writes 
> against the index. But it's definetely possible that the compound index 
> format makes this issue more visible or that it has its own bugs. In my 
> testing and in the initial testing of the Lucene dev team when we were 
> working on the compound index format we didn't see any such issues, but 
> of course that could just mean that we didn't look in the right place.

While I'm fairly certain our app fulfills all the locking requirements for lucene, it's possible 
that we've missed something as well. I'll go through the code again to see if there is any place I 
allow multiple threads access improperly.

> Anyway, it would be extremely helpful if you could produce a 
> self-contained program that demonstrates this behavior. I know that this 
> can be hard to impossible with concurrency-related issues, but still.

I'll see what I can do.

> Also, what OS is this running under? This issue is related to filesystem 
> locking behavior which is different between Unix and Windows.

Red Hat Linux release 9 (Shrike) under their 2.4.20-24.9smp kernel and Sun's 1.4.2_03 SDK. We did 
disable lucene's file based locking mechanism since our app is not setup to allow multiple different 
JVM's to write to a single index (one index per JVM) and we've had no end of issues with the file 
based locking. This was done by just setting FSDirectory.DISABLE_LOCKS to true;

> Does this happen every time or does it happen only under heavy load, etc.?
> What is the system doing when this occurs (i.e. indexing and searching 
> at the same time or just searching etc)?

Unfortunately, I have no answers for any of these questions. The best I can say is that the load on 
the server is usually very low (load average of 0.03) and the box itself has fast scsi based disks 
(IO is much faster for instance than my local development machine). The only other pertinent pience 
of info that I can think of is that this instance of the application was setup to update the index 
with new and updated content once every 10 minutes.


Regards,

Bruce Ritchie

> Bruce Ritchie wrote:
> 
>> We've started using the new compound indexing functionality available 
>> in lucene 1.3 final however we've been seeing an issue that I haven't 
>> quite resolved and was hoping that people on this list might be able 
>> to lend help on.
>>
>> The stack trace of the issue is:
>>
>> 2004.01.13 18:15 
>> [com.jivesoftware.forum.database.DbQuery.executeQuery(DbQuery.java:402)]
>> java.io.FileNotFoundException: /var/search/BfUfO9s1/_qb5.fnm (No such 
>> file or directory)
>>         at java.io.RandomAccessFile.open(Native Method)
>>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:204)
>>         at 
>> org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:392) 
>>
>>         at 
>> org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:421)
>>         at 
>> org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:294)
>>         at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:80)
>>         at 
>> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:138)
>>         at 
>> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:120)
>>         at 
>> org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:122)
>>         at org.apache.lucene.store.Lock$With.run(Lock.java:148)
>>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:111)
>>         at 
>> com.jivesoftware.forum.database.DbSearchManager.getSearcher(DbSearchManager.java:897) 
>>
>>         ...
>>
>> The error seems obvious enough from a quick glance of the above code - 
>> the file was been deleted. However, since we are using compound 
>> indexing there should be a /var/search/BfUfO9s1/_qb5.cfs file, however 
>> that file does not exist. The only files in the search directory are:
>>
>> -bash-2.05b$ ls -l
>> total 6908
>> -rw-r--r--    1 resin    js              4 Jan 13 18:14 deletable
>> -rw-r--r--    1 resin    js        7029055 Jan 13 10:03 _qau.cfs
>> -rw-r--r--    1 resin    js              9 Jan 13 18:14 _qb5.del
>> -rw-r--r--    1 resin    js          16060 Jan 13 18:14 _qb7.cfs
>> -rw-r--r--    1 resin    js             34 Jan 13 18:14 segments
>>
>>
>> From what I can tell it seems to me that the segments are somehow 
>> getting out of sync with what is actually on the filesystem somehow.
>>
>> Any suggestions are where to start looking for the cause of this bug?
>>
>>
>> Regards,
>>
>> Bruce Ritchie
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> 

-- 
AOL - bruceritchie101
ICQ - 9929791
MSN - bruce_ritchie101@hotmail.com
http://www.jivesoftware.com/

--------------ms090506090501080809090607
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIII/TCC
AtkwggJCoAMCAQICAwrLgDANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE
ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv
bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDMwOTIzMTkzNjM3WhcNMDQwOTIyMTkzNjM3
WjBIMR8wHQYDVQQDExZUaGF3dGUgRnJlZW1haWwgTWVtYmVyMSUwIwYJKoZIhvcNAQkBFhZi
cnVjZUBqaXZlc29mdHdhcmUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
ozJPHo8aUH2MrrLnFrmybY38uC8nlm26vBAPgGhSMthJlA0ShY9zAs4qGPyX1fIcANCKw0yP
QDESRRySYPcRYMBnwaeqtIwldTb1GTA6DAvZ/jpkM16DbAId17O86+4NeV7WoR+IguiyEUJq
IqecxdGiJsLi8xNM0rPan91sWTA+30kPvW12IU6MIa4XInPUazLqzZmA3CHoQJ2qorYCUx1c
P3VW2fbWMjpfis67MalWcFuBtThFuq5qsZMBEkFekoKPu2cpmjlKddrGj9+54U1CWBVQMl+9
HSB1/qhl1La8BYRhaZ5UG3c0za9yultXL9k1GxfZYVqDy2iMI5+ejQIDAQABozMwMTAhBgNV
HREEGjAYgRZicnVjZUBqaXZlc29mdHdhcmUuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcN
AQEEBQADgYEAEwi3SlYetSCzK5DLkClPMgTOM+P0qrIAFt+ar8uMZwr3xqFOxZlt6/qMHoXA
aOEWhZkHJ2dq8ebxjSdGXQF6/jKB42CEmYXmP6umb/2DBE6bVTy9aksnMnKeSyuqP1HWwlCf
5+Vk811biLueAupunISEh7P0bQXUgVtVVbOtAfIwggLZMIICQqADAgECAgMKy4AwDQYJKoZI
hvcNAQEEBQAwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQ
dHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENB
MB4XDTAzMDkyMzE5MzYzN1oXDTA0MDkyMjE5MzYzN1owSDEfMB0GA1UEAxMWVGhhd3RlIEZy
ZWVtYWlsIE1lbWJlcjElMCMGCSqGSIb3DQEJARYWYnJ1Y2VAaml2ZXNvZnR3YXJlLmNvbTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMyTx6PGlB9jK6y5xa5sm2N/LgvJ5Zt
urwQD4BoUjLYSZQNEoWPcwLOKhj8l9XyHADQisNMj0AxEkUckmD3EWDAZ8GnqrSMJXU29Rkw
OgwL2f46ZDNeg2wCHdezvOvuDXle1qEfiILoshFCaiKnnMXRoibC4vMTTNKz2p/dbFkwPt9J
D71tdiFOjCGuFyJz1Gsy6s2ZgNwh6ECdqqK2AlMdXD91Vtn21jI6X4rOuzGpVnBbgbU4Rbqu
arGTARJBXpKCj7tnKZo5SnXaxo/fueFNQlgVUDJfvR0gdf6oZdS2vAWEYWmeVBt3NM2vcrpb
Vy/ZNRsX2WFag8tojCOfno0CAwEAAaMzMDEwIQYDVR0RBBowGIEWYnJ1Y2VAaml2ZXNvZnR3
YXJlLmNvbTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUAA4GBABMIt0pWHrUgsyuQy5Ap
TzIEzjPj9KqyABbfmq/LjGcK98ahTsWZbev6jB6FwGjhFoWZBydnavHm8Y0nRl0Bev4ygeNg
hJmF5j+rpm/9gwROm1U8vWpLJzJynksrqj9R1sJQn+flZPNdW4i7ngLqbpyEhIez9G0F1IFb
VVWzrQHyMIIDPzCCAqigAwIBAgIBDTANBgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkEx
FTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFU
aGF3dGUgQ29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZp
c2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcN
AQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEz
MDcxNjIzNTk1OVowYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5n
IChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5n
IENBMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/
ox7svc31W/Iadr1/DDph8r9RzgHU5VAKMNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2
JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTXp6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wID
AQABo4GUMIGRMBIGA1UdEwEB/wQIMAYBAf8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDov
L2NybC50aGF3dGUuY29tL1RoYXd0ZVBlcnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQD
AgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG
9w0BAQUFAAOBgQBIjNFQg+oLLswNo2asZw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAg
k3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSeJVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghO
rvbqNOUQGls1TXfjViF4gtwhGTXeJLHTHUb/XV9lTzGCAzswggM3AgEBMGkwYjELMAkGA1UE
BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT
I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAgMKy4AwCQYFKw4DAhoFAKCC
AacwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMDQwMTE1MDYy
MTE2WjAjBgkqhkiG9w0BCQQxFgQUFOP6XXZ3kSurkSu0bZSEqPfVVW4wUgYJKoZIhvcNAQkP
MUUwQzAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4D
AgcwDQYIKoZIhvcNAwICASgweAYJKwYBBAGCNxAEMWswaTBiMQswCQYDVQQGEwJaQTElMCMG
A1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBl
cnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAwrLgDB6BgsqhkiG9w0BCRACCzFroGkwYjEL
MAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAq
BgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAgMKy4AwDQYJKoZI
hvcNAQEBBQAEggEAEzp7Yk1L3dRHkVrwGXpbLztqhJmXfWrKbMF2RAuH54brdaIgr1k6L04f
VOLIj+/qTyp5lVyw40wdUFTET+lXZIrYhYpi3kXiafQte2xW9qX9oG99EdMsnnfus4oBiCWf
ljGdxpo/6ocYNZx55/cRgRnUrV8NFOws4aDy9SZr3HXgdBCkbIMzujvHFZkFSAU8es+xZF3g
kRQQHrQM1hez9MjOV+K9kgqrxWF1M+95GLq3M4TJ4s1oD/svG41Xyr002Rql0PFzGMoxlZS+
MWIb7nrCoGsPCdQaAl9yYFGOoHTpqhQI9KG7YCjXY4dGkWtlLGNlSTyeL0T7JEb3P1FGEQAA
AAAAAA==
--------------ms090506090501080809090607--