lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Sitsky <s...@nuix.com>
Subject Re: java.io.IOException: background merge hit exception: Caused by: java.io.FileNotFoundException: _1vw.fdt
Date Wed, 28 Jul 2010 05:08:22 GMT
Incidentally.. this is what TerminateProcess does say from MSDN:

TerminateProcess initiates termination and returns immediately. This
stops execution of all threads within the process and requests
cancellation of all pending I/O. The terminated process cannot exit
until all pending I/O has been completed or canceled.

So I am not sure if this is a problem or not.. if a background merge
is happening while TerminateProcess() is called, then potentially, the
Lucene libraries may have written some data, but I wonder if the O/S
can "cancel" it before it is committed?  You would hope if fsync()
equivalent is called, that the pending I/O operations can't be
cancelled, but I do wonder?

On 28 July 2010 15:04, David Sitsky <sits@nuix.com> wrote:
> Hi Mike,
>
> Definitely no OOME (or other critical errors), and if there were, we
> would have terminated the program straight away.
>
> Cheers,
> David
>
> On 28 July 2010 01:44, Michael McCandless <lucene@mikemccandless.com> wrote:
>> Were there any exceptions during indexing, before the
>> TerminateProcess() call?  EG OOME?
>>
>> Mike
>>
>> On Tue, Jul 27, 2010 at 8:43 AM, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>>> On Mon, Jul 26, 2010 at 7:10 PM, David Sitsky <sits@nuix.com> wrote:
>>>> During processing.. there might be a number of reasons why we need to
>>>> shutdown the indexing process, but perhaps what is unusual is we call
>>>> the win32 API TerminateProcess() call rather than System.exit(), for
>>>> slightly obscure reasons.  When calling exit(), this still calls a
>>>> large body of code (for example dll shutdown hooks) which in some
>>>> situations, we found could "hang" the exiting process, which was a
>>>> problem for us.
>>>>
>>>> In a sense, this should be no different to killing the process under
>>>> Windows using the task manager, or kill -9 on unix systems.
>>>
>>> OK.  Lucene should be fine after a kill -9 or a TerminateProcess,
>>> assuming Windows really does act like Unix and any "committed" IO
>>> operations done by the process prior to the kill are in fact committed
>>> to the filesystem.
>>>
>>>> At no time did the machine itself crash, and the disk involved (I'm
>>>> told) was a local RAID filesystem.  I am guessing the disk has write
>>>> caching enabled, but given the machine didn't crash, this shouldn't
>>>> matter.
>>>
>>> Right, the write caching shouldn't matter since the machine didn't go down.
>>>
>>>> Something else that is slightly unusual is we explicitly call commit()
>>>> at certain times to flush indexing work to disk.
>>>
>>> OK that's fine.
>>>
>>>> Its interesting in both instances, CheckIndex said there was 1 broken
>>>> segment containing 1 document.
>>>
>>> Yeah that is curious.... I'll try to mull.
>>>
>>> Any chance you could run with an infoStream set on IndexWriter?  Then
>>> if this happens again I can pour over that...
>>>
>>> Mike
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
>
> --
> Cheers,
> David
>
> Nuix Pty Ltd
> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia    Ph: +61 2 9280 0699
> Web: http://www.nuix.com                            Fax: +61 2 9212 6902
>



-- 
Cheers,
David

Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia    Ph: +61 2 9280 0699
Web: http://www.nuix.com                            Fax: +61 2 9212 6902

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message