lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nadav Har'El" <...@il.ibm.com>
Subject Re: IndexWriter mergeSegments
Date Sun, 07 May 2006 14:11:39 GMT
Karel Tejnora <karel@tejnora.cz> wrote on 03/05/2006 02:29:06 AM:>
>..
> Before compound file is created the segments info are written but points
> to non-existing coumpound file then new .tmp is created and renamed to
.cfs
> Between time when new segments was written and new cfs is creating Index
> reader in different application whith no care about a lock can take
> segment with empty cfs resulting in empty hits.

Hi Karel, I've also been bothered by Lucene crash-intolerance bugs,
where killing Lucene in a specific (ill-)chosen moment leaves you
with an unusable index.

I filed a bug report in http://issues.apache.org/jira/browse/LUCENE-554
about another case where a crash can leave in an inconsistent state,
but it likely that the problem you reported is even more serious,
because it appears that the amount of time we are at risk is relatively
high in the case you reported, unlike in my bug report which requires
extraordinary bad luck to see.

> I see solution in swap those two blocks. firstly create final cfs and
> than write and delete old segments.
> Has anybody seen a problem with that?

Looking at the code, I see no reason why your suggested fix won't
work, and I don't see any negative side-effects. But I'm new to
Lucene so I hope someone with more experience can take a look.

By the way, I wonder why the current code first creates a file
called mergedName+".tmp", and only later renames it to ".cfs".
What's the point of doing that, when directory.renameFile() is
not atomic (see LUCENE-554), and in some cases it even resorts
to copying (see FSDirectory.renameFile()'s comments about "some jvms".)
And I also wonder why the existing code does the renameFile()
with the commit lock held, and not outside it.

> I can send patch but firstly I need to find svn client in gentoo :) and
> it's to late here.
> Can be smb so kind and give me link where I can find how to generate
> patch in lucene/apache way?

I'm sorry I can't really help you with that, I'm also new to Lucene
and now working on my first patch for it...
Good luck,
Nadav.

--
Nadav Har'El


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message