lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <>
Subject Re: ConcurrentMergeScheduler may spawn MergeThreads forever
Date Tue, 28 Apr 2009 13:27:24 GMT
> It's there so "anyUnhandledExceptions" can be called;

I will check the code again, but I remember that after commenting it, the
only compile errors I saw were from MergeThread adding the exception ...
perhaps I'm missing something, so I'll re-check the code.

I understand your point now - merging is an internal process to IW,
therefore there's no real user to notify on errors (e.g., even if IW knew
there is an error, what would it do exactly?), and I guess keep trying to
execute the merges is reasonable (while throwing the exceptions further - in
hope that the user code will catch those and do something with them).

BTW, what will happen if I encounter such exceptions, that are thrown
repeatedly, and shutdown the JVM? I guess the index will not be in a corrupt
state, right? The next time I'll open it, it should be in a state prior to
the merge, or at least prior to the merge that failed?

I think that sleeping in case of exceptions makes sense .. in case of IO
errors that are temporary, this will not spawn threads endlessly, and
sleeping will give an opportunity for the IO problem to resolve. In case of
bugs, which are supposed to be detected during test time, it should give the
developer a chance to kill the process relatively quickly.

On Tue, Apr 28, 2009 at 2:39 PM, Michael McCandless <> wrote:

> On Tue, Apr 28, 2009 at 8:28 AM, Shai Erera <> wrote:
> > Every merge hit the exception, yes.
> >
> > And actually, the exceptions list is not used anywhere besides MT adding
> the
> > exception to the list. That's why I was curious why it's there.
> It's there so "anyUnhandledExceptions" can be called; we could add a
> getter so that an app could query the CMS to see if there were
> exceptions?  But, an app can also subclass CMS & override
> handleMergeException to do something.
> > I still think we should protect this case somehow, because even if it
> hits a
> > disk-full exception, there's no point continuing to run infinitely?
> The disk full could clear up, eg if something external was trying to
> copy a massive file onto the same disk, IW could hit disk full, then
> the copy would fail and remove the partially copied massive file, and
> lots of space becomes available again.
> > So maybe
> > before spwaning the next thread, check the exceptions list and if it goes
> > over a certain threshold (10?) fail?
> But what does "fail" mean?  Stop doing any merges forever, for this IW
> instance?  That seems dangerous.  EG maybe the massive merge will
> fail, but little merges can proceed.  Also, stopping merging like this
> spooks me because you might see a few exceptions, but then they stop
> and you think all is good but in fact way too many segments are piling
> up.
> We already "fail" now, in that the thread that was doing the merge
> will throw an exception up to the JRE's default thread exception
> handler.  It's just that we then let MergePolicy select that merge
> again.
> I like the "sleep on exception" because 1) you will continue to see
> that exceptions are being thrown, but 2) it won't saturate your CPU.
> Mike
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message