Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jason.rutherglen@gmail.com
 designates 74.125.46.29 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=oVdRJhngNih31T/pzcReK3tqFIRB8Whlj/TqQJgGQpziaiBnV4vQIeGMTy5vpPiroL
         snHYzQZtuMlBneJIaPEIx5wlQizJWSL026PAzcc31HlBm8b7EQAmEP4y7Whrqqhz4gst
         9996F/pCz1qK/4AY/kSRbJ49ceNdEjgASolMo=
Message-ID: <85d3c3b60809260920x5ed78aerdefa89e25a5b8b59@mail.gmail.com>
Date: Fri, 26 Sep 2008 12:20:48 -0400
From: "Jason Rutherglen" <jason.rutherglen@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
In-Reply-To: <848FBF6C-4E8C-40FF-8A72-623778324F13@mikemccandless.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: 
 <0CD49D04A7B5534CA56257ADD9719EF7036AA1A3@indiaserv01.asite.asitehq.com>
	 <0B7F1A83-825C-406B-AAE6-23028E777DC0@mikemccandless.com>
	 <85d3c3b60809260858i66fb8debw19d65716f013a8ec@mail.gmail.com>
	 <848FBF6C-4E8C-40FF-8A72-623778324F13@mikemccandless.com>

I'm thinking more in terms of CRC32 checks performed on database
pages.  Is there a way to incorporate this technique in a way that
does not affect performance too much in Lucene?  The question is, when
is the CRC32 check is performed, and to which files is it applied if
any?

On Fri, Sep 26, 2008 at 12:13 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
>
> Corrupted data in what sense?
>
> EG if you don't trust your IO system to store data properly?
>
> Mike
>
> Jason Rutherglen wrote:
>
>> Mike,
>>
>> As part of my goal of trying to use Lucene as primary storage
>> mechanism (perhaps not the best idea), what do you think is the best
>> way to handle storing data in Lucene and preventing corrupted data the
>> way something like an SQL database handles corrupted data?  Or is
>> there simply no good way to do this?
>>
>> Jason
>>
>> On Fri, Sep 26, 2008 at 10:30 AM, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>>>
>>> It's perfectly fine to have a reader open on an index, while an
>>> IndexWriter
>>> runs optimize.
>>>
>>> Which version of Lucene are you using?  And which OS & filesystem?
>>>
>>> Mike
>>>
>>> Chaula Ganatra wrote:
>>>
>>>> It was the Reader on same index, which I did not close so gave exception
>>>> in writer.optimise()
>>>>
>>>> Chaula
>>>>
>>>> -----Original Message-----
>>>> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>>>> Sent: 26 September, 2008 7:17 PM
>>>> To: java-user@lucene.apache.org
>>>> Subject: Re: How to restore corrupted index
>>>>
>>>>
>>>> Can you post the full stack trace in both cases?
>>>>
>>>> Mike
>>>>
>>>> Chaula Ganatra wrote:
>>>>
>>>>> I found one case when such multiple files are remained, when we call
>>>>> writer.optimise() it throws exception and multiple files remained in
>>>>> index dir.
>>>>>
>>>>> After such multiple files, when we add document in index by calling
>>>>> writer.addDocument it throws java.lang.NegativeArraySizeException
>>>>>
>>>>> Regards,
>>>>> Chaula
>>>>>
>>>>> -----Original Message-----
>>>>> From: Grant Ingersoll [mailto:gsingers@apache.org]
>>>>> Sent: 26 September, 2008 6:02 PM
>>>>> To: java-user@lucene.apache.org
>>>>> Subject: Re: How to restore corrupted index
>>>>>
>>>>> There is the CheckIndex tool included in the distribution for
>>>>> checking/
>>>>> fixing bad indexes, but it can't solve everything.
>>>>>
>>>>> The bigger question is why it is happening to begin with.  Can you
>>>>> describe your indexing process?  How do you know the index is actually
>>>>> corrupted?  Are you seeing exceptions when opening it?
>>>>>
>>>>> -Grant
>>>>> On Sep 26, 2008, at 6:49 AM, Chaula Ganatra wrote:
>>>>>
>>>>>> We have an application in which index will be updated frequently.
>>>>>>
>>>>>> During development time, found that index files gets corrupted, i.e.
>>>>>> more than one  cfs files,some other extension files e.g. frq, fnm,
>>>>>> nrm
>>>>>>
>>>>>> Remains there in index directory.
>>>>>>
>>>>>> Is there any way that such issue does not occur at all or if it
>>>>>> happens
>>>>>> we can recover the index data again?
>>>>>>
>>>>>> It would be a great help, if some one can.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Chaula
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --------------------------
>>>>> Grant Ingersoll
>>>>> http://www.lucidimagination.com
>>>>>
>>>>> Lucene Helpful Hints:
>>>>> http://wiki.apache.org/lucene-java/BasicsOfPerformance
>>>>> http://wiki.apache.org/lucene-java/LuceneFAQ
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org