lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Sitsky <s...@nuix.com>
Subject Re: java.io.IOException: background merge hit exception: Caused by: java.io.FileNotFoundException: _1vw.fdt
Date Mon, 26 Jul 2010 23:10:43 GMT
During processing.. there might be a number of reasons why we need to
shutdown the indexing process, but perhaps what is unusual is we call
the win32 API TerminateProcess() call rather than System.exit(), for
slightly obscure reasons.  When calling exit(), this still calls a
large body of code (for example dll shutdown hooks) which in some
situations, we found could "hang" the exiting process, which was a
problem for us.

In a sense, this should be no different to killing the process under
Windows using the task manager, or kill -9 on unix systems.

At no time did the machine itself crash, and the disk involved (I'm
told) was a local RAID filesystem.  I am guessing the disk has write
caching enabled, but given the machine didn't crash, this shouldn't
matter.

Something else that is slightly unusual is we explicitly call commit()
at certain times to flush indexing work to disk.

Its interesting in both instances, CheckIndex said there was 1 broken
segment containing 1 document.

Cheers,
David

On 26 July 2010 22:24, Michael McCandless <lucene@mikemccandless.com> wrote:
> It's great that CheckIndex resolved it, but I'd like to get to the
> root cause if possible.
>
> Can you describe how the previous indexing process abnormally
> terminated?  Did the JRE crash/get killed?  Did the OS/machine crash /
> lose power?
>
> Abnormal termination of any kind, background merges running or not,
> should never cause this (unless we have a bug ;) ), except if fsync is
> not working correctly and the OS/machine had crashed.
>
> Mike
>
> On Mon, Jul 26, 2010 at 6:12 AM, David Sitsky <sits@nuix.com> wrote:
>> Running CheckIndex -fix fixed both indexes which is a relief.  In both
>> cases, it said 1 broken segment containing 1 document detected.
>>
>> So any ideas on what might have caused this in the first place?
>>
>> On 26 July 2010 16:58, David Sitsky <sits@nuix.com> wrote:
>>> As another data point, this happened on another index:
>>>
>>> java.io.IOException: background merge hit exception: _17a:C1003710
>>> _18h:C66391 _1ct:C45384 _1wt:C104479 _1zo:C23469 _271:C58570
>>> _2dk:C53760 _2h5:C109879 _2ik:C7713 _2ii:C121 _2ij:C243 _2il:C124
>>> _2im:C122 _2in:C115 _2io:C138 _2ip:C131 _2iq:C116 _2ir:C141 _2is:C183
>>> _2it:C147 _2iu:C129 _2iv:C143 _2iw:C111 _2ix:C123 _2iy:C448 _2iz:C157
>>> _2j0:C286 _2j1:C282 _2j2:C191 _2j3:C233 _2j4:C214 _2j5:C153 _2j6:C126
>>> _2j7:C166 _2j8:C411 _2j9:C122 _2ja:C94 _2jb:C2840 _2jc:C70 _2jd:C3
>>> _2je:C878 _2jf:C114 _2jg:C87 _2jh:C61 _2ji:C1 _2jj:C1 _2jk:C5 into
>>> _2jl [optimize] [mergeDocStores]
>>> ...
>>> Caused by: java.io.FileNotFoundException: D:\[snip]\_2ji.fdx (The
>>> system cannot find the file specified)
>>>
>>> We are going to try using the "check index" tool within Luke to see if
>>> this can help.
>>>
>>> In both instances, the Java program which has a single index writer
>>> instance did abnormally terminate, and automatically restarted to
>>> continue indexing.  Is it possible if the program abnormally
>>> terminated while a background merge was occurring that this happen?
>>> In our program, we explicitly control when data is committed.
>>>
>>> From what I can see, the program continued to add more documents to
>>> the index after the restart without issue, before doing the final
>>> optimise operation, which is where the above error comes from, like
>>> the trace I showed before.
>>>
>>> On 26 July 2010 14:39, David Sitsky <sits@nuix.com> wrote:
>>>> Correction - this was with Lucene 2.9.3.
>>>>
>>>> On 26 July 2010 14:21, David Sitsky <sits@nuix.com> wrote:
>>>>> Hi,
>>>>>
>>>>> A customer has been indexing a very large collection of documents that
>>>>> has been running over many days using 2.9.0.  During the optimisation
>>>>> stage, the following error occurred, and now the index can no longer
>>>>> be opened due to the "missing file".  I have been told this index is
>>>>> on a local RAID disk configuration on a Windows machine.  Is there
>>>>> anything we can do to somehow "recover" this index to make it useable?
>>>>>  Any help would be greatly appreciated.
>>>>>
>>>>> Is this possible this is a bug with 2.9.0?
>>>>>
>>>>> java.io.IOException: background merge hit exception: _8d:C110892
>>>>> _9s:C29548 _b7:C30930 _co:C20063 _e1:C43786 _fj:C102309 _gv:C67123
>>>>> _ia:C26329 _jr:C61859 _l5:C38116 _mj:C33069 _ny:C14417 _pe:C13849
>>>>> _qs:C21354 _s7:C10683 _u5:C11953 _v1:C12250 _wg:C18405 _xv:C24611
>>>>> _zg:C15294 _10p:C20039 _125:C24706 _15d:C17375 _176:C73888 _17r:C13939
>>>>> _19c:C9959 _1bz:C66884 _1dg:C6827 _1ex:C33196 _1g8:C16677 _1hn:C23601
>>>>> _1j2:C11191 _1kh:C9764 _1lw:C8585 _1nb:C24527 _1oq:C24600 _1q5:C23088
>>>>> _1rk:C10578 _1tl:C60504 _1ue:C23005 _1vu:C4571 _1vt:C6 _1vv:C6 _1vw:C1
>>>>> _1vx:C4 _1vy:C1 _1vz:C2 _1w0:C1 _1w1:C1 _1w2:C26 into _1w3 [optimize]
>>>>> [mergeDocStores]
>>>>>        at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2908)
>>>>>        at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2843)
>>>>>        at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2813)
>>>>>        [snip]
>>>>>        ...
>>>>> Caused by: java.io.FileNotFoundException: D:\[snip]\TextIndex\_1vw.fdt
>>>>> (The system cannot find the file specified)
>>>>>        at java.io.RandomAccessFile.open(Native Method)
>>>>>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>>>>>        at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
>>>>>        at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)
>>>>>        at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:65)
>>>>>        at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:104)
>>>>>        at org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:291)
>>>>>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:654)
>>>>>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:622)
>>>>>        at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:698)
>>>>>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5062)
>>>>>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4675)
>>>>>        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
>>>>>        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> David
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> David
>>>>
>>>> Nuix Pty Ltd
>>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia    Ph: +61 2 9280 0699
>>>> Web: http://www.nuix.com                            Fax: +61
2 9212 6902
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> David
>>>
>>> Nuix Pty Ltd
>>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia    Ph: +61 2 9280 0699
>>> Web: http://www.nuix.com                            Fax: +61 2
9212 6902
>>>
>>
>>
>>
>> --
>> Cheers,
>> David
>>
>> Nuix Pty Ltd
>> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia    Ph: +61 2 9280 0699
>> Web: http://www.nuix.com                            Fax: +61 2 9212
6902
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>



-- 
Cheers,
David

Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia    Ph: +61 2 9280 0699
Web: http://www.nuix.com                            Fax: +61 2 9212 6902

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message