lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: How to restore corrupted index
Date Fri, 26 Sep 2008 17:01:17 GMT

OK it does sound like you're primarily protecting against an  
untrustworthy storage system (or, maybe, Lucene bugs ;).

Probably the best option is to do this fully externally, ie, compute  
digest yourself, store it away in a separate Lucene field, then test  
the digest on loading the field?

Mike

Jason Rutherglen wrote:

> I'm thinking more in terms of CRC32 checks performed on database
> pages.  Is there a way to incorporate this technique in a way that
> does not affect performance too much in Lucene?  The question is, when
> is the CRC32 check is performed, and to which files is it applied if
> any?
>
> On Fri, Sep 26, 2008 at 12:13 PM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
>>
>> Corrupted data in what sense?
>>
>> EG if you don't trust your IO system to store data properly?
>>
>> Mike
>>
>> Jason Rutherglen wrote:
>>
>>> Mike,
>>>
>>> As part of my goal of trying to use Lucene as primary storage
>>> mechanism (perhaps not the best idea), what do you think is the best
>>> way to handle storing data in Lucene and preventing corrupted data  
>>> the
>>> way something like an SQL database handles corrupted data?  Or is
>>> there simply no good way to do this?
>>>
>>> Jason
>>>
>>> On Fri, Sep 26, 2008 at 10:30 AM, Michael McCandless
>>> <lucene@mikemccandless.com> wrote:
>>>>
>>>> It's perfectly fine to have a reader open on an index, while an
>>>> IndexWriter
>>>> runs optimize.
>>>>
>>>> Which version of Lucene are you using?  And which OS & filesystem?
>>>>
>>>> Mike
>>>>
>>>> Chaula Ganatra wrote:
>>>>
>>>>> It was the Reader on same index, which I did not close so gave  
>>>>> exception
>>>>> in writer.optimise()
>>>>>
>>>>> Chaula
>>>>>
>>>>> -----Original Message-----
>>>>> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>>>>> Sent: 26 September, 2008 7:17 PM
>>>>> To: java-user@lucene.apache.org
>>>>> Subject: Re: How to restore corrupted index
>>>>>
>>>>>
>>>>> Can you post the full stack trace in both cases?
>>>>>
>>>>> Mike
>>>>>
>>>>> Chaula Ganatra wrote:
>>>>>
>>>>>> I found one case when such multiple files are remained, when we 

>>>>>> call
>>>>>> writer.optimise() it throws exception and multiple files  
>>>>>> remained in
>>>>>> index dir.
>>>>>>
>>>>>> After such multiple files, when we add document in index by  
>>>>>> calling
>>>>>> writer.addDocument it throws java.lang.NegativeArraySizeException
>>>>>>
>>>>>> Regards,
>>>>>> Chaula
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Grant Ingersoll [mailto:gsingers@apache.org]
>>>>>> Sent: 26 September, 2008 6:02 PM
>>>>>> To: java-user@lucene.apache.org
>>>>>> Subject: Re: How to restore corrupted index
>>>>>>
>>>>>> There is the CheckIndex tool included in the distribution for
>>>>>> checking/
>>>>>> fixing bad indexes, but it can't solve everything.
>>>>>>
>>>>>> The bigger question is why it is happening to begin with.  Can  
>>>>>> you
>>>>>> describe your indexing process?  How do you know the index is  
>>>>>> actually
>>>>>> corrupted?  Are you seeing exceptions when opening it?
>>>>>>
>>>>>> -Grant
>>>>>> On Sep 26, 2008, at 6:49 AM, Chaula Ganatra wrote:
>>>>>>
>>>>>>> We have an application in which index will be updated  
>>>>>>> frequently.
>>>>>>>
>>>>>>> During development time, found that index files gets  
>>>>>>> corrupted, i.e.
>>>>>>> more than one  cfs files,some other extension files e.g. frq,
 
>>>>>>> fnm,
>>>>>>> nrm
>>>>>>>
>>>>>>> Remains there in index directory.
>>>>>>>
>>>>>>> Is there any way that such issue does not occur at all or if
it
>>>>>>> happens
>>>>>>> we can recover the index data again?
>>>>>>>
>>>>>>> It would be a great help, if some one can.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Chaula
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --------------------------
>>>>>> Grant Ingersoll
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>> Lucene Helpful Hints:
>>>>>> http://wiki.apache.org/lucene-java/BasicsOfPerformance
>>>>>> http://wiki.apache.org/lucene-java/LuceneFAQ
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message