lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nader S. Henein" <...@bayt.net>
Subject RE: Disappearing segments
Date Mon, 03 May 2004 04:29:52 GMT
You're catching an exception and acting on it, but you're not reporting it,
for now, comment out the deletion and copyfrombackup and try reporting
errors, if the batch is failing on a regular basis you want to know about
it, comment the deletion and copy across code out, also watch out that if
you backup the index during an indexing you could end up with a limp index
missing a few files, hence the missing segments, I would check for write and
commit locks pre-backup so as to avoid that. This is probably caused by two
unrelated errors first batchindex() fails then the backup restores a version
that may not have all the indexes there (depending when it was backed up)
thereby giving you the feeling that segments are disappearing randomly.

Hope this helps.

Nader Henein

-----Original Message-----
From: Kelvin Tan [mailto:lists-kelvin@relevanz.com] 
Sent: Monday, May 03, 2004 6:52 AM
To: Lucene Users List
Subject: RE: Disappearing segments


Thanks for responding Nader.

hmmmm...you've hit the nail on the spot. I do have a cron job which backs up
the 
index. Its run in a batch index scheduled job. 

The logic is basically

backupindex()
try
{
	batchindex()
}
catch(Exception e)
{
	deleteindex();
	copyfrombackuptoindex()
	deletebackup();
}

I assume that the original index before backing up was complete and
'working'. 
I'm also deleting the index that failed, instead of just overwriting. Where
did 
I go wrong? 

I'm not checking that the index isn't write-locked before backing up, but I 
don't think that's the problem (though it very well can be a separate
problem).

Kelvin

On Fri, 30 Apr 2004 23:20:42 +0400, Nader Henein said:
> Could you share you're indexing code, and just to make sure id there 
> anything running on your machine that could delete these files, like 
> an a cron job that'll back up the index.
> 
> You could go by process of elimination and shut down your server and 
> see if the files disappear, coz if the problem is contained within the 
> server you know that you can safely go on the DEBUG rampage.
> 
> Nader
> 
> -----Original Message-----
> From: Kelvin Tan [mailto:lists-kelvin@relevanz.com]
> Sent: Friday, April 30, 2004 9:15 AM
> To: Lucene Users List
> Subject: Re: Disappearing segments
> 
> An update:
> 
> Daniel Naber suggested using IndexWriter.setUseCompoundFile() to see 
> if it happens with the compound index format. Before I had a chance to 
> try it out, this happened:
> 
> java.io.FileNotFoundException: C:\index\segments (The system cannot 
> find the file specified) at java.io.RandomAccessFile.open(Native 
> Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:200)
> at
> org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.j
> ava:321)
> at
> org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:329)
> at
> org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:268)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:71)
> at
> org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:154)
> at org.apache.lucene.store.Lock$With.run(Lock.java:116)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:149)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:131)
> 
> so even the segments file somehow got deleted. Hoping someone can shed 
> some light on this...
> 
> Kelvin
> 
> On Thu, 29 Apr 2004 11:45:36 +0800, Kelvin Tan said:
>> Errr, sorry for the cross-post to lucene-dev as well, but I realized 
>> this mail really belongs on lucene-user...
>> 
>> I've been experiencing intermittent disappearing segments which 
>> result in the following stacktrace:
>> 
>> Caused by: java.io.FileNotFoundException: C:\index\_1ae.fnm (The 
>> system cannot find the file specified) at 
>> java.io.RandomAccessFile.open(Native Method) at
>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:200)
>> at 
>> org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.j
>> a
>> va:321) at
>> org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:329)
>> at org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:268)
>> at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:78)
>> at
>> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:104)
>> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:95)
>> at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:112)
>> at org.apache.lucene.store.Lock$With.run(Lock.java:116)
>> at org.apache.lucene.index.IndexReader.open(IndexReader.java:103)
>> at org.apache.lucene.index.IndexReader.open(IndexReader.java:91)
>> at
>> org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:75)
>> 
>> The segment that disappears (_1ae.fnm) varies.
>> 
>> I can't seem to reproduce this error consistently, so don't have a 
>> clue what might cause it, but it usually happens after the 
>> application has been running for some time. Has anyone experienced 
>> something similar, or can anyone point
> me
>> in the right direction?
>> 
>> When this occurs, I need to rebuild the entire index for it to be 
>> usable. Very troubling indeed...
>> 
>> Kelvin
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org





---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message